==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-6b9df475cd-csxlf Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-9d1e5395-43zw/10.142.0.28 Start Time: Wed, 24 Apr 2024 05:20:36 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=6b9df475cd skaffold.dev/run-id=737e6a69-d863-4e69-8f57-d28ee8754f0b Annotations: Status: Running IP: 10.106.42.136 IPs: IP: 10.106.42.136 Controlled By: ReplicaSet/lodemon-6b9df475cd Containers: lodemon: Container ID: containerd://3771703fc28b5054fbbc330fb3cf574539acacc6d9285b20cfdd58d5e1f3a082 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:211ddeee1073a9baac07b9c978705770ae4867030362650bc93603cb72ebfbe1 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Wed, 24 Apr 2024 05:20:37 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h6c77 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-h6c77: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 10:24:41 INFO 10:24:41 INFO [loop_until]: kubectl --namespace=xlou top pods 10:24:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:24:41 INFO [loop_until]: OK (rc = 0) 10:24:41 DEBUG --- stdout --- 10:24:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 9m 3466Mi am-769b98f8d5-2rr6f 9m 3407Mi am-769b98f8d5-h2r87 10m 3432Mi ds-cts-0 5m 561Mi ds-cts-1 6m 538Mi ds-cts-2 5m 502Mi ds-idrepo-0 4226m 11789Mi ds-idrepo-1 1041m 6995Mi ds-idrepo-2 17m 5817Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4246m 3954Mi idm-6c95d7588f-j6s6c 3801m 3561Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 332m 664Mi 10:24:41 DEBUG --- stderr --- 10:24:41 DEBUG 10:24:45 INFO 10:24:45 INFO [loop_until]: kubectl --namespace=xlou top node 10:24:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:24:45 INFO [loop_until]: OK (rc = 0) 10:24:45 DEBUG --- stdout --- 10:24:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 76m 0% 8228Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 1137m 7% 8387Mi 14% gke-xlou-cdm-default-pool-9d1e5395-jnd9 3999m 25% 13089Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 58m 0% 1747Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 342m 2% 6192Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 56m 0% 1735Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 55m 0% 1644Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 82m 0% 4984Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 3650m 22% 5516Mi 9% gke-xlou-cdm-ds-174e6abc-k8cj 141m 0% 2465Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 72m 0% 4833Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 62m 0% 1185Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 4744m 29% 4848Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 372m 2% 4091Mi 6% 10:24:45 DEBUG --- stderr --- 10:24:45 DEBUG 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713950730" 10:25:30 ERROR [http_cmd]: ERROR 10:25:30 ERROR --- status code --- 10:25:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:25:30 ERROR --- http response --- 10:25:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:25:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "instance": "10.142.0.24:9100" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "instance": "10.142.0.74:9100" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "12235.766666666666" 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "pod": "ds-cts-1" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "pod": "ds-cts-0" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "0" 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:25:30 DEBUG "pod": "ds-idrepo-1" 10:25:30 DEBUG }, 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG [...] 10:25:30 DEBUG [...] 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:30 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:25:30 DEBUG "pod": "ds-idrepo-0" 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG }, 10:25:30 DEBUG }, 10:25:30 DEBUG { 10:25:30 DEBUG { 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "status": "success", 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG 1713950730, 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG "data": { 10:25:30 DEBUG "data": { 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG "0" 10:25:30 DEBUG "389.7" 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG ] 10:25:30 DEBUG ] 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG "result": [ 10:25:30 DEBUG "result": [ 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG { 10:25:30 DEBUG { 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:30 DEBUG ] 10:25:30 DEBUG ] 10:25:30 DEBUG "metric": { 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:30 DEBUG "metric": { 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG "pod": "ds-cts-1" 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:25:30 DEBUG { 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG }, 10:25:30 DEBUG { 10:25:30 DEBUG }, 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "data": { 10:25:30 DEBUG 10:25:30 DEBUG "data": { 10:25:30 DEBUG 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG 10:25:30 DEBUG "result": [ 10:25:30 DEBUG 10:25:30 DEBUG "result": [ 10:25:30 DEBUG 10:25:30 DEBUG { 10:25:30 DEBUG [...] 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 DEBUG { 10:25:30 DEBUG [...] 10:25:30 DEBUG "metric": { 10:25:30 DEBUG 10:25:30 DEBUG "metric": { 10:25:30 DEBUG 10:25:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:25:30 DEBUG 10:25:30 DEBUG "instance": "10.142.0.24:9100" 10:25:30 DEBUG 10:25:30 DEBUG }, 10:25:30 DEBUG 10:25:30 DEBUG }, 10:25:30 DEBUG 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "pod": "ds-cts-0" 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:25:30 DEBUG 10:25:30 DEBUG }, 10:25:30 DEBUG 10:25:30 DEBUG }, 10:25:30 DEBUG 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG [...] 10:25:30 DEBUG "0" 10:25:30 DEBUG [...] 10:25:30 DEBUG "0" 10:25:30 DEBUG 10:25:30 DEBUG ] 10:25:30 DEBUG 10:25:30 DEBUG ] 10:25:30 DEBUG 10:25:30 DEBUG } 10:25:30 DEBUG 10:25:30 DEBUG } 10:25:30 DEBUG 10:25:30 DEBUG ] 10:25:30 DEBUG 10:25:30 DEBUG ] 10:25:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:25:30 DEBUG } 10:25:30 DEBUG "instance": "10.142.0.74:9100" 10:25:30 DEBUG } 10:25:30 DEBUG }, 10:25:30 DEBUG } 10:25:30 DEBUG }, 10:25:30 DEBUG } 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "0" 10:25:30 DEBUG "0.00023333333333338166" 10:25:30 DEBUG ] 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "0" 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713950730" 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "instance": "10.142.0.24:9100" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "instance": "10.142.0.74:9100" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "8916.8" 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "0.3356066782040445" 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "0" 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 DEBUG --- http response --- 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG { 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG { 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:25:30 DEBUG "data": { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "data": { 10:25:30 DEBUG { 10:25:30 DEBUG "result": [ 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG { 10:25:30 DEBUG "result": [ 10:25:30 DEBUG "data": { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "result": [ 10:25:30 DEBUG }, 10:25:30 DEBUG "instance": "10.142.0.24:9100" 10:25:30 DEBUG { 10:25:30 DEBUG "value": [ 10:25:30 DEBUG }, 10:25:30 DEBUG "metric": { 10:25:30 DEBUG 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "pod": "ds-idrepo-1" 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG }, 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "value": [ 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG }, 10:25:30 DEBUG "instance": "10.142.0.74:9100" 10:25:30 DEBUG 10:25:30 DEBUG "value": [ 10:25:30 DEBUG }, 10:25:30 DEBUG 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "pod": "ds-cts-0" 10:25:30 DEBUG "0" 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG }, 10:25:30 DEBUG ] 10:25:30 DEBUG "2884412416" 10:25:30 DEBUG "value": [ 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG 1713950730, 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG "0" 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG ] 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG --- http response --- 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG } 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG } 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG { 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:30 DEBUG "data": { 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG { 10:25:30 DEBUG { 10:25:30 DEBUG "result": [ 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG { 10:25:30 DEBUG "data": { 10:25:30 DEBUG "data": { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:25:30 DEBUG "result": [ 10:25:30 DEBUG "result": [ 10:25:30 DEBUG }, 10:25:30 DEBUG { 10:25:30 DEBUG { 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG 10:25:30 DEBUG "pod": "ds-idrepo-1" 10:25:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:25:30 DEBUG 10:25:30 DEBUG }, 10:25:30 DEBUG }, 10:25:30 DEBUG 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "value": [ 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG [...] 10:25:30 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG }, 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "pod": "ds-idrepo-0" 10:25:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:25:30 DEBUG "437.71878354203926" 10:25:30 DEBUG }, 10:25:30 DEBUG }, 10:25:30 DEBUG ] 10:25:30 DEBUG "value": [ 10:25:30 DEBUG "value": [ 10:25:30 DEBUG } 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG ] 10:25:30 DEBUG "389.73333333333335" 10:25:30 DEBUG "0.015" 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "pod": "ds-cts-1" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "pod": "ds-cts-0" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "0" 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "0" 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950730" 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "pod": "ds-idrepo-1" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "pod": "ds-idrepo-0" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "389.7" 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713950730" 10:25:30 INFO [http_cmd]: http status code OK 10:25:30 DEBUG --- status code --- 10:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:30 DEBUG --- http response --- 10:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:30 DEBUG ----- output ----- 10:25:30 DEBUG { 10:25:30 DEBUG "status": "success", 10:25:30 DEBUG "data": { 10:25:30 DEBUG "resultType": "vector", 10:25:30 DEBUG "result": [ 10:25:30 DEBUG { 10:25:30 DEBUG "metric": { 10:25:30 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG [...] 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG 10:25:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:25:30 DEBUG }, 10:25:30 DEBUG "value": [ 10:25:30 DEBUG 1713950730, 10:25:30 DEBUG "836136960" 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG ] 10:25:30 DEBUG } 10:25:30 DEBUG } 10:25:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950731" 10:25:31 INFO [http_cmd]: http status code OK 10:25:31 DEBUG --- status code --- 10:25:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:31 DEBUG --- http response --- 10:25:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:25:31 DEBUG ----- output ----- 10:25:31 DEBUG { 10:25:31 DEBUG "status": "success", 10:25:31 DEBUG "data": { 10:25:31 DEBUG "resultType": "vector", 10:25:31 DEBUG "result": [ 10:25:31 DEBUG { 10:25:31 DEBUG "metric": { 10:25:31 DEBUG "instance": "10.142.0.24:9100" 10:25:31 DEBUG }, 10:25:31 DEBUG "value": [ 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG [...] 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG "instance": "10.142.0.74:9100" 10:25:31 DEBUG }, 10:25:31 DEBUG "value": [ 10:25:31 DEBUG 1713950731, 10:25:31 DEBUG "0" 10:25:31 DEBUG ] 10:25:31 DEBUG } 10:25:31 DEBUG ] 10:25:31 DEBUG } 10:25:31 DEBUG } 10:25:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950731" 10:25:31 INFO [http_cmd]: http status code OK 10:25:31 DEBUG --- status code --- 10:25:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:31 DEBUG --- http response --- 10:25:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:31 DEBUG ----- output ----- 10:25:31 DEBUG { 10:25:31 DEBUG "status": "success", 10:25:31 DEBUG "data": { 10:25:31 DEBUG "resultType": "vector", 10:25:31 DEBUG "result": [ 10:25:31 DEBUG { 10:25:31 DEBUG "metric": { 10:25:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:25:31 DEBUG }, 10:25:31 DEBUG "value": [ 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG [...] 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:25:31 DEBUG }, 10:25:31 DEBUG "value": [ 10:25:31 DEBUG 1713950731, 10:25:31 DEBUG "0" 10:25:31 DEBUG ] 10:25:31 DEBUG } 10:25:31 DEBUG ] 10:25:31 DEBUG } 10:25:31 DEBUG } 10:25:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950731" 10:25:31 INFO [http_cmd]: http status code OK 10:25:31 DEBUG --- status code --- 10:25:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:31 DEBUG --- http response --- 10:25:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:25:31 DEBUG ----- output ----- 10:25:31 DEBUG { 10:25:31 DEBUG "status": "success", 10:25:31 DEBUG "data": { 10:25:31 DEBUG "resultType": "vector", 10:25:31 DEBUG "result": [ 10:25:31 DEBUG { 10:25:31 DEBUG "metric": { 10:25:31 DEBUG "pod": "ds-idrepo-1" 10:25:31 DEBUG }, 10:25:31 DEBUG "value": [ 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG [...] 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG 10:25:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:25:31 DEBUG }, 10:25:31 DEBUG "value": [ 10:25:31 DEBUG 1713950731, 10:25:31 DEBUG "0" 10:25:31 DEBUG ] 10:25:31 DEBUG } 10:25:31 DEBUG ] 10:25:31 DEBUG } 10:25:31 DEBUG } 10:25:41 INFO 10:25:41 INFO [loop_until]: kubectl --namespace=xlou top pods 10:25:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:25:41 INFO [loop_until]: OK (rc = 0) 10:25:41 DEBUG --- stdout --- 10:25:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 6m 3465Mi am-769b98f8d5-2rr6f 6m 3408Mi am-769b98f8d5-h2r87 5m 3432Mi ds-cts-0 4m 561Mi ds-cts-1 6m 538Mi ds-cts-2 6m 502Mi ds-idrepo-0 4341m 11790Mi ds-idrepo-1 454m 7756Mi ds-idrepo-2 12m 5817Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 6459m 4278Mi idm-6c95d7588f-j6s6c 2932m 3561Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 321m 704Mi 10:25:41 DEBUG --- stderr --- 10:25:41 DEBUG 10:25:45 INFO 10:25:45 INFO [loop_until]: kubectl --namespace=xlou top node 10:25:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:25:45 INFO [loop_until]: OK (rc = 0) 10:25:45 DEBUG --- stdout --- 10:25:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 71m 0% 8236Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 839m 5% 9385Mi 16% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4333m 27% 13099Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 64m 0% 1760Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 348m 2% 6169Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 60m 0% 1737Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 59m 0% 1638Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 93m 0% 4963Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 5114m 32% 5828Mi 9% gke-xlou-cdm-ds-174e6abc-k8cj 131m 0% 2443Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 72m 0% 4828Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 65m 0% 1189Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 2757m 17% 4864Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 382m 2% 4131Mi 7% 10:25:45 DEBUG --- stderr --- 10:25:45 DEBUG 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713950790" 10:26:30 ERROR [http_cmd]: ERROR 10:26:30 ERROR --- status code --- 10:26:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:26:30 ERROR --- http response --- 10:26:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:26:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950790" 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG { 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG "data": { 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "result": [ 10:26:30 DEBUG { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "instance": "10.142.0.24:9100" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG [...] 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG "instance": "10.142.0.74:9100" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG "14602.666666666666" 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950790" 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG { 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG "data": { 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "result": [ 10:26:30 DEBUG { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "instance": "10.142.0.24:9100" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG [...] 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG "instance": "10.142.0.74:9100" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG "0.0001666666666666335" 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG { 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG "data": { 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "result": [ 10:26:30 DEBUG { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "pod": "ds-cts-1" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG [...] 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG "pod": "ds-cts-0" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG "0" 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG { 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG "data": { 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 DEBUG { 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG "result": [ 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 DEBUG "data": { 10:26:30 DEBUG { 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "result": [ 10:26:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:26:30 DEBUG { 10:26:30 DEBUG }, 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "value": [ 10:26:30 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:26:30 DEBUG 10:26:30 DEBUG }, 10:26:30 DEBUG 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG [...] 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG [...] 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:26:30 DEBUG 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG }, 10:26:30 DEBUG "pod": "ds-cts-0" 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG "value": [ 10:26:30 DEBUG }, 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG "0" 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG ] 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:30 DEBUG "0" 10:26:30 DEBUG } 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG ] 10:26:30 DEBUG ] 10:26:30 DEBUG { 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG "data": { 10:26:30 DEBUG } 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG } 10:26:30 DEBUG "result": [ 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 DEBUG { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG [...] 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG "0" 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG } 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG { 10:26:30 DEBUG { 10:26:30 DEBUG { 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG "data": { 10:26:30 DEBUG "data": { 10:26:30 DEBUG "data": { 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "result": [ 10:26:30 DEBUG "result": [ 10:26:30 DEBUG "result": [ 10:26:30 DEBUG { 10:26:30 DEBUG { 10:26:30 DEBUG { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "pod": "ds-cts-1" 10:26:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:26:30 DEBUG "pod": "ds-idrepo-1" 10:26:30 DEBUG }, 10:26:30 DEBUG }, 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG "value": [ 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG [...] 10:26:30 DEBUG [...] 10:26:30 DEBUG [...] 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG "pod": "ds-cts-0" 10:26:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG "pod": "ds-idrepo-0" 10:26:30 DEBUG }, 10:26:30 DEBUG }, 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG "value": [ 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG "0" 10:26:30 DEBUG "0" 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:30 DEBUG "573.8666666666667" 10:26:30 DEBUG ] 10:26:30 DEBUG ] 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 DEBUG { 10:26:30 DEBUG } 10:26:30 DEBUG ] 10:26:30 DEBUG ] 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 DEBUG "data": { 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG } 10:26:30 DEBUG "result": [ 10:26:30 DEBUG { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG [...] 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG "pod": "ds-cts-0" 10:26:30 DEBUG }, 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950790" 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG "0" 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950790" 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713950790" 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG { 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG "data": { 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "result": [ 10:26:30 DEBUG { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "instance": "10.142.0.24:9100" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG [...] 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG "instance": "10.142.0.74:9100" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG "12324.033333333335" 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 DEBUG ----- output ----- 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG { 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG "data": { 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:30 DEBUG "result": [ 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG { 10:26:30 DEBUG { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG "instance": "10.142.0.24:9100" 10:26:30 DEBUG "data": { 10:26:30 DEBUG }, 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "value": [ 10:26:30 DEBUG "result": [ 10:26:30 DEBUG 10:26:30 DEBUG { 10:26:30 DEBUG 10:26:30 DEBUG "metric": { 10:26:30 DEBUG 10:26:30 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:26:30 DEBUG [...] 10:26:30 DEBUG }, 10:26:30 DEBUG 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG "instance": "10.142.0.74:9100" 10:26:30 DEBUG 10:26:30 DEBUG }, 10:26:30 DEBUG [...] 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG 10:26:30 DEBUG "2884668416" 10:26:30 DEBUG 10:26:30 DEBUG ] 10:26:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:26:30 DEBUG } 10:26:30 DEBUG }, 10:26:30 DEBUG ] 10:26:30 DEBUG "value": [ 10:26:30 DEBUG } 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG } 10:26:30 DEBUG "0.33678707963500415" 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 INFO [http_cmd]: http status code OK 10:26:30 DEBUG --- status code --- 10:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:30 DEBUG --- http response --- 10:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:30 DEBUG ----- output ----- 10:26:30 DEBUG { 10:26:30 DEBUG "status": "success", 10:26:30 DEBUG "data": { 10:26:30 DEBUG "resultType": "vector", 10:26:30 DEBUG "result": [ 10:26:30 DEBUG { 10:26:30 DEBUG "metric": { 10:26:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG [...] 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG 10:26:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:26:30 DEBUG }, 10:26:30 DEBUG "value": [ 10:26:30 DEBUG 1713950790, 10:26:30 DEBUG "0" 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG ] 10:26:30 DEBUG } 10:26:30 DEBUG } 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713950790" 10:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950791" 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 DEBUG --- status code --- 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 DEBUG --- status code --- 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG --- status code --- 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 DEBUG { 10:26:31 DEBUG { 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG --- status code --- 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG { 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG "data": { 10:26:31 DEBUG "data": { 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG "data": { 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:31 DEBUG "result": [ 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 DEBUG "result": [ 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG { 10:26:31 DEBUG --- status code --- 10:26:31 DEBUG { 10:26:31 DEBUG "result": [ 10:26:31 DEBUG { 10:26:31 DEBUG "metric": { 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG "metric": { 10:26:31 DEBUG { 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG "pod": "ds-idrepo-1" 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG "pod": "ds-idrepo-1" 10:26:31 DEBUG "metric": { 10:26:31 DEBUG "data": { 10:26:31 DEBUG }, 10:26:31 DEBUG }, 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG "value": [ 10:26:31 DEBUG "value": [ 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG }, 10:26:31 DEBUG "result": [ 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG { 10:26:31 DEBUG "value": [ 10:26:31 DEBUG { 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG 10:26:31 DEBUG "metric": { 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "data": { 10:26:31 DEBUG 10:26:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:26:31 DEBUG [...] 10:26:31 DEBUG [...] 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG 10:26:31 DEBUG }, 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "result": [ 10:26:31 DEBUG [...] 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG { 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "metric": { 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "pod": "ds-idrepo-0" 10:26:31 DEBUG "pod": "ds-cts-0" 10:26:31 DEBUG "pod": "ds-idrepo-1" 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG }, 10:26:31 DEBUG }, 10:26:31 DEBUG }, 10:26:31 DEBUG "pod": "idm-6c95d7588f-j6s6c" 10:26:31 DEBUG [...] 10:26:31 DEBUG "value": [ 10:26:31 DEBUG "value": [ 10:26:31 DEBUG "value": [ 10:26:31 DEBUG }, 10:26:31 DEBUG 10:26:31 DEBUG 1713950790, 10:26:31 DEBUG 1713950790, 10:26:31 DEBUG 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 10:26:31 DEBUG "573.8" 10:26:31 DEBUG "0" 10:26:31 DEBUG 10:26:31 DEBUG 1713950790, 10:26:31 DEBUG 10:26:31 DEBUG ] 10:26:31 DEBUG ] 10:26:31 DEBUG 10:26:31 DEBUG "922.4071011129543" 10:26:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:26:31 DEBUG } 10:26:31 DEBUG } 10:26:31 DEBUG [...] 10:26:31 DEBUG ] 10:26:31 DEBUG }, 10:26:31 DEBUG ] 10:26:31 DEBUG ] 10:26:31 DEBUG 10:26:31 DEBUG } 10:26:31 DEBUG "value": [ 10:26:31 DEBUG } 10:26:31 DEBUG } 10:26:31 DEBUG 10:26:31 DEBUG ] 10:26:31 DEBUG 1713950790, 10:26:31 DEBUG } 10:26:31 DEBUG } 10:26:31 DEBUG 10:26:31 DEBUG } 10:26:31 DEBUG "0.015" 10:26:31 DEBUG "pod": "ds-idrepo-0" 10:26:31 DEBUG } 10:26:31 DEBUG ] 10:26:31 DEBUG }, 10:26:31 DEBUG } 10:26:31 DEBUG "value": [ 10:26:31 DEBUG ] 10:26:31 DEBUG 1713950791, 10:26:31 DEBUG } 10:26:31 DEBUG "573.9" 10:26:31 DEBUG } 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG } 10:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950791" 10:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950791" 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 DEBUG --- status code --- 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG { 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG "data": { 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG "result": [ 10:26:31 DEBUG { 10:26:31 DEBUG "metric": { 10:26:31 DEBUG "pod": "ds-cts-1" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG [...] 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "pod": "ds-cts-0" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 1713950791, 10:26:31 DEBUG "0" 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG } 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 DEBUG --- status code --- 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG { 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG "data": { 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG "result": [ 10:26:31 DEBUG { 10:26:31 DEBUG "metric": { 10:26:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG [...] 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 1713950791, 10:26:31 DEBUG "0" 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG } 10:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713950791" 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 DEBUG --- status code --- 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG { 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG "data": { 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG "result": [ 10:26:31 DEBUG { 10:26:31 DEBUG "metric": { 10:26:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG [...] 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 1713950791, 10:26:31 DEBUG "884936704" 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG } 10:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950791" 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 DEBUG --- status code --- 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG { 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG "data": { 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG "result": [ 10:26:31 DEBUG { 10:26:31 DEBUG "metric": { 10:26:31 DEBUG "instance": "10.142.0.24:9100" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG [...] 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "instance": "10.142.0.74:9100" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 1713950791, 10:26:31 DEBUG "0" 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG } 10:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950791" 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 DEBUG --- status code --- 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG { 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG "data": { 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG "result": [ 10:26:31 DEBUG { 10:26:31 DEBUG "metric": { 10:26:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG [...] 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 1713950791, 10:26:31 DEBUG "0" 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG } 10:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950791" 10:26:31 INFO [http_cmd]: http status code OK 10:26:31 DEBUG --- status code --- 10:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:26:31 DEBUG --- http response --- 10:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:26:31 DEBUG ----- output ----- 10:26:31 DEBUG { 10:26:31 DEBUG "status": "success", 10:26:31 DEBUG "data": { 10:26:31 DEBUG "resultType": "vector", 10:26:31 DEBUG "result": [ 10:26:31 DEBUG { 10:26:31 DEBUG "metric": { 10:26:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG [...] 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG 10:26:31 DEBUG "pod": "idm-6c95d7588f-j6s6c" 10:26:31 DEBUG }, 10:26:31 DEBUG "value": [ 10:26:31 DEBUG 1713950791, 10:26:31 DEBUG "0" 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG ] 10:26:31 DEBUG } 10:26:31 DEBUG } 10:26:41 INFO 10:26:41 INFO [loop_until]: kubectl --namespace=xlou top pods 10:26:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:26:41 INFO [loop_until]: OK (rc = 0) 10:26:41 DEBUG --- stdout --- 10:26:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3465Mi am-769b98f8d5-2rr6f 4m 3407Mi am-769b98f8d5-h2r87 8m 3432Mi ds-cts-0 4m 561Mi ds-cts-1 5m 538Mi ds-cts-2 5m 502Mi ds-idrepo-0 3799m 11787Mi ds-idrepo-1 48m 8326Mi ds-idrepo-2 14m 5817Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4290m 4282Mi idm-6c95d7588f-j6s6c 3225m 3788Mi lodemon-6b9df475cd-csxlf 5m 68Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 264m 752Mi 10:26:41 DEBUG --- stderr --- 10:26:41 DEBUG 10:26:46 INFO 10:26:46 INFO [loop_until]: kubectl --namespace=xlou top node 10:26:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:26:46 INFO [loop_until]: OK (rc = 0) 10:26:46 DEBUG --- stdout --- 10:26:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 75m 0% 8239Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 646m 4% 9792Mi 16% gke-xlou-cdm-default-pool-9d1e5395-jnd9 3846m 24% 13080Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 58m 0% 1748Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 342m 2% 6146Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 58m 0% 1737Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1636Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 81m 0% 4968Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 3932m 24% 5834Mi 9% gke-xlou-cdm-ds-174e6abc-k8cj 151m 0% 2470Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 63m 0% 4829Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 61m 0% 1192Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3654m 22% 5084Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 325m 2% 4181Mi 7% 10:26:46 DEBUG --- stderr --- 10:26:46 DEBUG 10:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713950850" 10:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950850" 10:27:30 ERROR [http_cmd]: ERROR 10:27:30 ERROR --- status code --- 10:27:30 INFO [http_cmd]: http status code OK 10:27:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:27:30 DEBUG --- status code --- 10:27:30 ERROR --- http response --- 10:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:27:30 DEBUG --- http response --- 10:27:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:30 DEBUG ----- output ----- 10:27:30 DEBUG { 10:27:30 DEBUG "status": "success", 10:27:30 DEBUG "data": { 10:27:30 DEBUG "resultType": "vector", 10:27:30 DEBUG "result": [ 10:27:30 DEBUG { 10:27:30 DEBUG "metric": { 10:27:30 DEBUG "instance": "10.142.0.24:9100" 10:27:30 DEBUG }, 10:27:30 DEBUG "value": [ 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG [...] 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG "instance": "10.142.0.74:9100" 10:27:30 DEBUG }, 10:27:30 DEBUG "value": [ 10:27:30 DEBUG 1713950850, 10:27:30 DEBUG "13305.966666666665" 10:27:30 DEBUG ] 10:27:30 DEBUG } 10:27:30 DEBUG ] 10:27:30 DEBUG } 10:27:30 DEBUG } 10:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950850" 10:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950850" 10:27:30 INFO [http_cmd]: http status code OK 10:27:30 INFO [http_cmd]: http status code OK 10:27:30 DEBUG --- status code --- 10:27:30 DEBUG --- status code --- 10:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:30 DEBUG --- http response --- 10:27:30 DEBUG --- http response --- 10:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:30 DEBUG ----- output ----- 10:27:30 DEBUG ----- output ----- 10:27:30 DEBUG { 10:27:30 DEBUG { 10:27:30 DEBUG "status": "success", 10:27:30 DEBUG "status": "success", 10:27:30 DEBUG "data": { 10:27:30 DEBUG "data": { 10:27:30 DEBUG "resultType": "vector", 10:27:30 DEBUG "resultType": "vector", 10:27:30 DEBUG "result": [ 10:27:30 DEBUG "result": [ 10:27:30 DEBUG { 10:27:30 DEBUG { 10:27:30 DEBUG "metric": { 10:27:30 DEBUG "metric": { 10:27:30 DEBUG "pod": "ds-cts-1" 10:27:30 DEBUG "instance": "10.142.0.24:9100" 10:27:30 DEBUG }, 10:27:30 DEBUG }, 10:27:30 DEBUG "value": [ 10:27:30 DEBUG "value": [ 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG [...] 10:27:30 DEBUG [...] 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG "pod": "ds-cts-0" 10:27:30 DEBUG "instance": "10.142.0.74:9100" 10:27:30 DEBUG }, 10:27:30 DEBUG }, 10:27:30 DEBUG "value": [ 10:27:30 DEBUG "value": [ 10:27:30 DEBUG 1713950850, 10:27:30 DEBUG 1713950850, 10:27:30 DEBUG "0" 10:27:30 DEBUG "0.0002666666666666373" 10:27:30 DEBUG ] 10:27:30 DEBUG ] 10:27:30 DEBUG } 10:27:30 DEBUG } 10:27:30 DEBUG ] 10:27:30 DEBUG ] 10:27:30 DEBUG } 10:27:30 DEBUG } 10:27:30 DEBUG } 10:27:30 DEBUG } 10:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950850" 10:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950850" 10:27:30 INFO [http_cmd]: http status code OK 10:27:30 DEBUG --- status code --- 10:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:30 DEBUG --- http response --- 10:27:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:27:30 INFO [http_cmd]: http status code OK 10:27:30 DEBUG --- status code --- 10:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:30 DEBUG --- http response --- 10:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:30 DEBUG ----- output ----- 10:27:30 DEBUG { 10:27:30 DEBUG "status": "success", 10:27:30 DEBUG "data": { 10:27:30 DEBUG "resultType": "vector", 10:27:30 DEBUG "result": [ 10:27:30 DEBUG { 10:27:30 DEBUG "metric": { 10:27:30 DEBUG "pod": "am-769b98f8d5-h2r87" 10:27:30 DEBUG }, 10:27:30 DEBUG "value": [ 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG [...] 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG 10:27:30 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:27:30 DEBUG }, 10:27:30 DEBUG "value": [ 10:27:30 DEBUG 1713950850, 10:27:30 DEBUG "0" 10:27:30 DEBUG ] 10:27:30 DEBUG } 10:27:30 DEBUG ] 10:27:30 DEBUG } 10:27:30 DEBUG } 10:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950850" 10:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950850" 10:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950850" 10:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950850" 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950850, 10:27:31 DEBUG "0" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "pod": "ds-cts-1" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "pod": "ds-cts-0" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950850, 10:27:31 DEBUG "0" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "instance": "10.142.0.24:9100" 10:27:31 DEBUG }, 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG "value": [ 10:27:31 DEBUG --- status code --- 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 DEBUG 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 DEBUG 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 DEBUG [...] 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:27:31 DEBUG 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG 10:27:31 DEBUG { 10:27:31 DEBUG 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "instance": "10.142.0.74:9100" 10:27:31 DEBUG "data": { 10:27:31 DEBUG }, 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "value": [ 10:27:31 DEBUG "result": [ 10:27:31 DEBUG 1713950850, 10:27:31 DEBUG { 10:27:31 DEBUG "2884875264" 10:27:31 DEBUG "metric": { 10:27:31 DEBUG ] 10:27:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:27:31 DEBUG } 10:27:31 DEBUG }, 10:27:31 DEBUG ] 10:27:31 DEBUG "value": [ 10:27:31 DEBUG } 10:27:31 DEBUG 10:27:31 DEBUG } 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950850, 10:27:31 DEBUG "3.4562622498039275" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:27:31 DEBUG "pod": "ds-idrepo-1" 10:27:31 DEBUG }, 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG 10:27:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:27:31 DEBUG "pod": "ds-idrepo-0" 10:27:31 DEBUG }, 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG { 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "0" 10:27:31 DEBUG "data": { 10:27:31 DEBUG "409.0333333333333" 10:27:31 DEBUG ] 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG "result": [ 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950851" 10:27:31 DEBUG ] 10:27:31 DEBUG { 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG "metric": { 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:27:31 DEBUG } 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "73464.58894534047" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "instance": "10.142.0.24:9100" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "instance": "10.142.0.74:9100" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "9852.466666666667" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713950851" 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG { 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:31 DEBUG { 10:27:31 DEBUG ----- output ----- 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG "metric": { 10:27:31 DEBUG { 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG }, 10:27:31 DEBUG "data": { 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG "value": [ 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:31 DEBUG "result": [ 10:27:31 DEBUG 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG [...] 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:27:31 DEBUG 10:27:31 DEBUG "data": { 10:27:31 DEBUG }, 10:27:31 DEBUG 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG "result": [ 10:27:31 DEBUG 10:27:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:27:31 DEBUG { 10:27:31 DEBUG 10:27:31 DEBUG }, 10:27:31 DEBUG "metric": { 10:27:31 DEBUG 10:27:31 DEBUG "value": [ 10:27:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:27:31 DEBUG [...] 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG }, 10:27:31 DEBUG 10:27:31 DEBUG "558.6660756299656" 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG ] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG } 10:27:31 DEBUG 10:27:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:27:31 DEBUG ] 10:27:31 DEBUG 10:27:31 DEBUG }, 10:27:31 DEBUG } 10:27:31 DEBUG [...] 10:27:31 DEBUG "value": [ 10:27:31 DEBUG } 10:27:31 DEBUG 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG 10:27:31 DEBUG "0.22603356824039406" 10:27:31 DEBUG 10:27:31 DEBUG ] 10:27:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:27:31 DEBUG } 10:27:31 DEBUG }, 10:27:31 DEBUG ] 10:27:31 DEBUG "value": [ 10:27:31 DEBUG } 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG } 10:27:31 DEBUG "0.03333333333333333" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "0.03333333333333333" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "0.015" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "pod": "ds-cts-1" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "pod": "ds-cts-0" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "0" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "pod": "ds-idrepo-1" 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "pod": "ds-idrepo-1" 10:27:31 DEBUG }, 10:27:31 DEBUG "pod": "ds-idrepo-1" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "pod": "ds-idrepo-0" 10:27:31 DEBUG 10:27:31 DEBUG "pod": "ds-idrepo-0" 10:27:31 DEBUG }, 10:27:31 DEBUG "pod": "ds-cts-0" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "409.0333333333333" 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "409" 10:27:31 DEBUG ] 10:27:31 DEBUG "0" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713950851" 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950851" 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "889339904" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "instance": "10.142.0.24:9100" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "instance": "10.142.0.74:9100" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "0.00033333333333333365" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "0" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950851" 10:27:31 INFO [http_cmd]: http status code OK 10:27:31 DEBUG --- status code --- 10:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:27:31 DEBUG --- http response --- 10:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:27:31 DEBUG ----- output ----- 10:27:31 DEBUG { 10:27:31 DEBUG "status": "success", 10:27:31 DEBUG "data": { 10:27:31 DEBUG "resultType": "vector", 10:27:31 DEBUG "result": [ 10:27:31 DEBUG { 10:27:31 DEBUG "metric": { 10:27:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG [...] 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG 10:27:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:27:31 DEBUG }, 10:27:31 DEBUG "value": [ 10:27:31 DEBUG 1713950851, 10:27:31 DEBUG "0" 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG ] 10:27:31 DEBUG } 10:27:31 DEBUG } 10:27:41 INFO 10:27:41 INFO [loop_until]: kubectl --namespace=xlou top pods 10:27:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:27:41 INFO [loop_until]: OK (rc = 0) 10:27:41 DEBUG --- stdout --- 10:27:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 5m 3466Mi am-769b98f8d5-2rr6f 5m 3407Mi am-769b98f8d5-h2r87 9m 3432Mi ds-cts-0 4m 560Mi ds-cts-1 6m 538Mi ds-cts-2 5m 502Mi ds-idrepo-0 3558m 11795Mi ds-idrepo-1 713m 8135Mi ds-idrepo-2 9m 5817Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 3493m 4509Mi idm-6c95d7588f-j6s6c 4067m 3791Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 315m 788Mi 10:27:41 DEBUG --- stderr --- 10:27:41 DEBUG 10:27:46 INFO 10:27:46 INFO [loop_until]: kubectl --namespace=xlou top node 10:27:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:27:46 INFO [loop_until]: OK (rc = 0) 10:27:46 DEBUG --- stdout --- 10:27:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 69m 0% 8234Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 760m 4% 9435Mi 16% gke-xlou-cdm-default-pool-9d1e5395-jnd9 3829m 24% 13100Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 62m 0% 1755Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 356m 2% 6173Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 56m 0% 1734Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 57m 0% 1642Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 82m 0% 4972Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 3526m 22% 6070Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 141m 0% 2445Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 66m 0% 4836Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 59m 0% 1180Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 4761m 29% 5098Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 358m 2% 4236Mi 7% 10:27:46 DEBUG --- stderr --- 10:27:46 DEBUG 10:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950910" 10:28:30 INFO [http_cmd]: http status code OK 10:28:30 DEBUG --- status code --- 10:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:30 DEBUG --- http response --- 10:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:28:30 DEBUG ----- output ----- 10:28:30 DEBUG { 10:28:30 DEBUG "status": "success", 10:28:30 DEBUG "data": { 10:28:30 DEBUG "resultType": "vector", 10:28:30 DEBUG "result": [ 10:28:30 DEBUG { 10:28:30 DEBUG "metric": { 10:28:30 DEBUG "instance": "10.142.0.24:9100" 10:28:30 DEBUG }, 10:28:30 DEBUG "value": [ 10:28:30 DEBUG 10:28:30 DEBUG 10:28:30 DEBUG 10:28:30 DEBUG [...] 10:28:30 DEBUG 10:28:30 DEBUG 10:28:30 DEBUG 10:28:30 DEBUG "instance": "10.142.0.74:9100" 10:28:30 DEBUG }, 10:28:30 DEBUG "value": [ 10:28:30 DEBUG 1713950910, 10:28:30 DEBUG "12151.366666666667" 10:28:30 DEBUG ] 10:28:30 DEBUG } 10:28:30 DEBUG ] 10:28:30 DEBUG } 10:28:30 DEBUG } 10:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713950910" 10:28:30 ERROR [http_cmd]: ERROR 10:28:30 ERROR --- status code --- 10:28:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:28:30 ERROR --- http response --- 10:28:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:28:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950910" 10:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950910" 10:28:30 INFO [http_cmd]: http status code OK 10:28:30 DEBUG --- status code --- 10:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:30 DEBUG --- http response --- 10:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:28:30 DEBUG ----- output ----- 10:28:30 DEBUG { 10:28:30 DEBUG "status": "success", 10:28:30 DEBUG "data": { 10:28:30 DEBUG "resultType": "vector", 10:28:30 DEBUG "result": [ 10:28:30 DEBUG { 10:28:30 DEBUG "metric": { 10:28:30 DEBUG "pod": "ds-cts-1" 10:28:30 DEBUG }, 10:28:30 DEBUG "value": [ 10:28:30 DEBUG 10:28:30 DEBUG 10:28:30 DEBUG 10:28:30 DEBUG [...] 10:28:30 DEBUG 10:28:30 DEBUG 10:28:30 DEBUG 10:28:30 DEBUG "pod": "ds-cts-0" 10:28:30 DEBUG }, 10:28:30 DEBUG "value": [ 10:28:30 DEBUG 1713950910, 10:28:30 DEBUG "0" 10:28:30 DEBUG ] 10:28:30 DEBUG } 10:28:30 DEBUG ] 10:28:30 DEBUG } 10:28:30 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950911" 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG { 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:28:31 DEBUG { 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG "metric": { 10:28:31 DEBUG { 10:28:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG }, 10:28:31 DEBUG "data": { 10:28:31 DEBUG "value": [ 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG 10:28:31 DEBUG "result": [ 10:28:31 DEBUG 10:28:31 DEBUG { 10:28:31 DEBUG 10:28:31 DEBUG "metric": { 10:28:31 DEBUG [...] 10:28:31 DEBUG "instance": "10.142.0.24:9100" 10:28:31 DEBUG 10:28:31 DEBUG }, 10:28:31 DEBUG 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:28:31 DEBUG 10:28:31 DEBUG }, 10:28:31 DEBUG 10:28:31 DEBUG "value": [ 10:28:31 DEBUG [...] 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG 10:28:31 DEBUG "0" 10:28:31 DEBUG 10:28:31 DEBUG ] 10:28:31 DEBUG 10:28:31 DEBUG } 10:28:31 DEBUG "instance": "10.142.0.74:9100" 10:28:31 DEBUG ] 10:28:31 DEBUG }, 10:28:31 DEBUG } 10:28:31 DEBUG "value": [ 10:28:31 DEBUG } 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0.00030000000000001136" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "pod": "ds-idrepo-1" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "29576170.971110344" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "pod": "ds-idrepo-1" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950911" 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "pod": "ds-cts-1" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "ds-cts-0" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 DEBUG "data": { 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG { 10:28:31 DEBUG "result": [ 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG { 10:28:31 DEBUG "data": { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "pod": "ds-idrepo-1" 10:28:31 DEBUG "result": [ 10:28:31 DEBUG }, 10:28:31 DEBUG { 10:28:31 DEBUG "value": [ 10:28:31 DEBUG "metric": { 10:28:31 DEBUG 10:28:31 DEBUG "instance": "10.142.0.24:9100" 10:28:31 DEBUG 10:28:31 DEBUG }, 10:28:31 DEBUG 10:28:31 DEBUG "value": [ 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:28:31 DEBUG 10:28:31 DEBUG }, 10:28:31 DEBUG 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "instance": "10.142.0.74:9100" 10:28:31 DEBUG "0.25887468914026956" 10:28:31 DEBUG }, 10:28:31 DEBUG ] 10:28:31 DEBUG "value": [ 10:28:31 DEBUG } 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG ] 10:28:31 DEBUG "2885092352" 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "pod": "ds-idrepo-1" 10:28:31 DEBUG "instance": "10.142.0.24:9100" 10:28:31 DEBUG }, 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "ds-idrepo-0" 10:28:31 DEBUG "instance": "10.142.0.74:9100" 10:28:31 DEBUG }, 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "312" 10:28:31 DEBUG "8929.066666666668" 10:28:31 DEBUG ] 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713950911" 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0.27700517587626866" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG "data": { 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG "result": [ 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG { 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:28:31 DEBUG "metric": { 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:28:31 DEBUG { 10:28:31 DEBUG }, 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "value": [ 10:28:31 DEBUG "data": { 10:28:31 DEBUG 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG 10:28:31 DEBUG "result": [ 10:28:31 DEBUG 10:28:31 DEBUG { 10:28:31 DEBUG [...] 10:28:31 DEBUG "metric": { 10:28:31 DEBUG 10:28:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:28:31 DEBUG 10:28:31 DEBUG }, 10:28:31 DEBUG 10:28:31 DEBUG "value": [ 10:28:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:28:31 DEBUG 10:28:31 DEBUG }, 10:28:31 DEBUG 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG [...] 10:28:31 DEBUG "0" 10:28:31 DEBUG 10:28:31 DEBUG ] 10:28:31 DEBUG 10:28:31 DEBUG } 10:28:31 DEBUG 10:28:31 DEBUG ] 10:28:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:28:31 DEBUG } 10:28:31 DEBUG }, 10:28:31 DEBUG } 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0.015" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "pod": "ds-cts-1" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "ds-cts-0" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:28:31 DEBUG "data": { 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG { 10:28:31 DEBUG "result": [ 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG { 10:28:31 DEBUG "data": { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "pod": "ds-idrepo-1" 10:28:31 DEBUG "result": [ 10:28:31 DEBUG }, 10:28:31 DEBUG { 10:28:31 DEBUG "value": [ 10:28:31 DEBUG "metric": { 10:28:31 DEBUG 10:28:31 DEBUG "pod": "ds-idrepo-1" 10:28:31 DEBUG 10:28:31 DEBUG }, 10:28:31 DEBUG 10:28:31 DEBUG "value": [ 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG "pod": "ds-idrepo-0" 10:28:31 DEBUG 10:28:31 DEBUG }, 10:28:31 DEBUG 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "pod": "ds-idrepo-0" 10:28:31 DEBUG "312" 10:28:31 DEBUG }, 10:28:31 DEBUG ] 10:28:31 DEBUG "value": [ 10:28:31 DEBUG } 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG ] 10:28:31 DEBUG "312" 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950911" 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG { 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:28:31 DEBUG "data": { 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG { 10:28:31 DEBUG "result": [ 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG { 10:28:31 DEBUG "data": { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "pod": "ds-idrepo-1" 10:28:31 DEBUG "result": [ 10:28:31 DEBUG }, 10:28:31 DEBUG { 10:28:31 DEBUG "value": [ 10:28:31 DEBUG "metric": { 10:28:31 DEBUG 10:28:31 DEBUG "instance": "10.142.0.24:9100" 10:28:31 DEBUG 10:28:31 DEBUG }, 10:28:31 DEBUG 10:28:31 DEBUG "value": [ 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG "pod": "ds-cts-0" 10:28:31 DEBUG 10:28:31 DEBUG }, 10:28:31 DEBUG 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "instance": "10.142.0.74:9100" 10:28:31 DEBUG "0" 10:28:31 DEBUG }, 10:28:31 DEBUG ] 10:28:31 DEBUG "value": [ 10:28:31 DEBUG } 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG ] 10:28:31 DEBUG "0" 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "945848320" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950911" 10:28:31 INFO [http_cmd]: http status code OK 10:28:31 DEBUG --- status code --- 10:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:28:31 DEBUG --- http response --- 10:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:28:31 DEBUG ----- output ----- 10:28:31 DEBUG { 10:28:31 DEBUG "status": "success", 10:28:31 DEBUG "data": { 10:28:31 DEBUG "resultType": "vector", 10:28:31 DEBUG "result": [ 10:28:31 DEBUG { 10:28:31 DEBUG "metric": { 10:28:31 DEBUG "pod": "ds-idrepo-1" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG [...] 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG 10:28:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:28:31 DEBUG }, 10:28:31 DEBUG "value": [ 10:28:31 DEBUG 1713950911, 10:28:31 DEBUG "0" 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG ] 10:28:31 DEBUG } 10:28:31 DEBUG } 10:28:41 INFO 10:28:41 INFO [loop_until]: kubectl --namespace=xlou top pods 10:28:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:28:41 INFO [loop_until]: OK (rc = 0) 10:28:41 DEBUG --- stdout --- 10:28:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 5m 3465Mi am-769b98f8d5-2rr6f 4m 3408Mi am-769b98f8d5-h2r87 6m 3432Mi ds-cts-0 6m 561Mi ds-cts-1 5m 538Mi ds-cts-2 5m 502Mi ds-idrepo-0 3638m 11797Mi ds-idrepo-1 267m 8260Mi ds-idrepo-2 10m 5817Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4632m 4508Mi idm-6c95d7588f-j6s6c 3038m 3957Mi lodemon-6b9df475cd-csxlf 4m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 272m 809Mi 10:28:41 DEBUG --- stderr --- 10:28:41 DEBUG 10:28:46 INFO 10:28:46 INFO [loop_until]: kubectl --namespace=xlou top node 10:28:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:28:46 INFO [loop_until]: OK (rc = 0) 10:28:46 DEBUG --- stdout --- 10:28:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 71m 0% 8246Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 197m 1% 9541Mi 16% gke-xlou-cdm-default-pool-9d1e5395-jnd9 3635m 22% 13093Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 62m 0% 1753Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 344m 2% 6128Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1730Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 57m 0% 1637Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 83m 0% 4973Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4255m 26% 6070Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 128m 0% 2453Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 68m 0% 4827Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 62m 0% 1193Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 2490m 15% 5254Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 330m 2% 4237Mi 7% 10:28:46 DEBUG --- stderr --- 10:28:46 DEBUG 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "instance": "10.142.0.24:9100" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "instance": "10.142.0.74:9100" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "12290" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 ERROR [http_cmd]: ERROR 10:29:31 ERROR --- status code --- 10:29:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:29:31 ERROR --- http response --- 10:29:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:29:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "ds-cts-1" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "ds-cts-0" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:29:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG { 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "data": { 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG "result": [ 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG { 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:29:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:29:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:29:31 DEBUG }, 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 DEBUG }, 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG "value": [ 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG [...] 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG { 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "data": { 10:29:31 DEBUG 10:29:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:29:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "pod": "ds-cts-0" 10:29:31 DEBUG }, 10:29:31 DEBUG }, 10:29:31 DEBUG "result": [ 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG "value": [ 10:29:31 DEBUG { 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "metric": { 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG "0" 10:29:31 DEBUG "instance": "10.142.0.24:9100" 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG ] 10:29:31 DEBUG }, 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 DEBUG "value": [ 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG ] 10:29:31 DEBUG 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 DEBUG 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 DEBUG 10:29:31 DEBUG } 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "instance": "10.142.0.74:9100" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0.00013333333333337785" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "ds-cts-1" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "ds-cts-0" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "instance": "10.142.0.24:9100" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "instance": "10.142.0.74:9100" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "9143.600000000002" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "ds-cts-0" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "ds-idrepo-1" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "ds-idrepo-0" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "528.9" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "instance": "10.142.0.24:9100" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "instance": "10.142.0.74:9100" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "2885475328" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0.28293835275625223" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "ds-idrepo-1" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "ds-idrepo-0" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "528.9333333333333" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0.015" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "ds-cts-1" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "ds-cts-0" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713950971" 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:29:31 DEBUG }, 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "946724864" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "ds-idrepo-1" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "ds-idrepo-0" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "528.9" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "instance": "10.142.0.24:9100" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "instance": "10.142.0.74:9100" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "ds-idrepo-1" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "ds-cts-0" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713950971" 10:29:31 INFO [http_cmd]: http status code OK 10:29:31 DEBUG --- status code --- 10:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:29:31 DEBUG --- http response --- 10:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:29:31 DEBUG ----- output ----- 10:29:31 DEBUG { 10:29:31 DEBUG "status": "success", 10:29:31 DEBUG "data": { 10:29:31 DEBUG "resultType": "vector", 10:29:31 DEBUG "result": [ 10:29:31 DEBUG { 10:29:31 DEBUG "metric": { 10:29:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG [...] 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG 10:29:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:29:31 DEBUG }, 10:29:31 DEBUG "value": [ 10:29:31 DEBUG 1713950971, 10:29:31 DEBUG "0" 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG ] 10:29:31 DEBUG } 10:29:31 DEBUG } 10:29:41 INFO 10:29:41 INFO [loop_until]: kubectl --namespace=xlou top pods 10:29:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:29:41 INFO [loop_until]: OK (rc = 0) 10:29:41 DEBUG --- stdout --- 10:29:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 6m 3407Mi am-769b98f8d5-h2r87 7m 3432Mi ds-cts-0 5m 562Mi ds-cts-1 6m 540Mi ds-cts-2 5m 502Mi ds-idrepo-0 1570m 11798Mi ds-idrepo-1 2213m 8806Mi ds-idrepo-2 7m 5817Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 2556m 4510Mi idm-6c95d7588f-j6s6c 3270m 3960Mi lodemon-6b9df475cd-csxlf 1m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 235m 814Mi 10:29:41 DEBUG --- stderr --- 10:29:41 DEBUG 10:29:46 INFO 10:29:46 INFO [loop_until]: kubectl --namespace=xlou top node 10:29:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:29:46 INFO [loop_until]: OK (rc = 0) 10:29:46 DEBUG --- stdout --- 10:29:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 70m 0% 8235Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 2286m 14% 10027Mi 17% gke-xlou-cdm-default-pool-9d1e5395-jnd9 1625m 10% 13107Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 60m 0% 1762Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 353m 2% 6184Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 60m 0% 1733Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 57m 0% 1639Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 83m 0% 4972Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 3611m 22% 6151Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 146m 0% 2462Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 67m 0% 4824Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 62m 0% 1192Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3722m 23% 5264Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 317m 1% 4241Mi 7% 10:29:46 DEBUG --- stderr --- 10:29:46 DEBUG 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "instance": "10.142.0.24:9100" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "instance": "10.142.0.74:9100" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "12381.066666666668" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951031" 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 ERROR [http_cmd]: ERROR 10:30:31 ERROR --- status code --- 10:30:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:30:31 ERROR --- http response --- 10:30:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:30:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:30:31 DEBUG { 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG { 10:30:31 DEBUG "data": { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "result": [ 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG { 10:30:31 DEBUG "result": [ 10:30:31 DEBUG "metric": { 10:30:31 DEBUG { 10:30:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:30:31 DEBUG "metric": { 10:30:31 DEBUG }, 10:30:31 DEBUG "pod": "ds-idrepo-1" 10:30:31 DEBUG "value": [ 10:30:31 DEBUG }, 10:30:31 DEBUG 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:30:31 DEBUG 10:30:31 DEBUG }, 10:30:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:30:31 DEBUG "value": [ 10:30:31 DEBUG }, 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG "0" 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG ] 10:30:31 DEBUG "20153.548032479703" 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "ds-cts-1" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "ds-cts-0" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG { 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:30:31 DEBUG { 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG "metric": { 10:30:31 DEBUG { 10:30:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG }, 10:30:31 DEBUG "data": { 10:30:31 DEBUG "value": [ 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG 10:30:31 DEBUG "result": [ 10:30:31 DEBUG 10:30:31 DEBUG { 10:30:31 DEBUG 10:30:31 DEBUG "metric": { 10:30:31 DEBUG [...] 10:30:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:30:31 DEBUG 10:30:31 DEBUG }, 10:30:31 DEBUG 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:30:31 DEBUG 10:30:31 DEBUG }, 10:30:31 DEBUG 10:30:31 DEBUG "value": [ 10:30:31 DEBUG [...] 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG 10:30:31 DEBUG "0" 10:30:31 DEBUG 10:30:31 DEBUG ] 10:30:31 DEBUG 10:30:31 DEBUG } 10:30:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:30:31 DEBUG ] 10:30:31 DEBUG }, 10:30:31 DEBUG } 10:30:31 DEBUG "value": [ 10:30:31 DEBUG } 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "ds-idrepo-1" 10:30:31 DEBUG "instance": "10.142.0.24:9100" 10:30:31 DEBUG }, 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:30:31 DEBUG "instance": "10.142.0.74:9100" 10:30:31 DEBUG }, 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "424.2852217364148" 10:30:31 DEBUG "9165.966666666667" 10:30:31 DEBUG ] 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "instance": "10.142.0.24:9100" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "instance": "10.142.0.74:9100" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0.0002666666666667557" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "ds-cts-1" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "ds-cts-0" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951031" 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "ds-idrepo-1" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "1.476617995540241" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "instance": "10.142.0.24:9100" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "instance": "10.142.0.74:9100" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "2885690368" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951031" 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "ds-idrepo-1" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "ds-idrepo-0" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "499.76666666666665" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0.3431104539436107" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "ds-idrepo-1" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "ds-idrepo-0" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "499.73333333333335" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0.015" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "ds-cts-1" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "ds-cts-0" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "ds-idrepo-1" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "1017524224" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "ds-idrepo-1" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "ds-idrepo-0" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "499.73333333333335" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951031" 10:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951031" 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "instance": "10.142.0.24:9100" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "instance": "10.142.0.74:9100" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0.0006666666666666673" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:31 INFO [http_cmd]: http status code OK 10:30:31 DEBUG --- status code --- 10:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:30:31 DEBUG --- http response --- 10:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:30:31 DEBUG ----- output ----- 10:30:31 DEBUG { 10:30:31 DEBUG "status": "success", 10:30:31 DEBUG "data": { 10:30:31 DEBUG "resultType": "vector", 10:30:31 DEBUG "result": [ 10:30:31 DEBUG { 10:30:31 DEBUG "metric": { 10:30:31 DEBUG "pod": "ds-idrepo-1" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG [...] 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG 10:30:31 DEBUG "pod": "ds-cts-0" 10:30:31 DEBUG }, 10:30:31 DEBUG "value": [ 10:30:31 DEBUG 1713951031, 10:30:31 DEBUG "0" 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG ] 10:30:31 DEBUG } 10:30:31 DEBUG } 10:30:41 INFO 10:30:41 INFO [loop_until]: kubectl --namespace=xlou top pods 10:30:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:30:42 INFO [loop_until]: OK (rc = 0) 10:30:42 DEBUG --- stdout --- 10:30:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 8m 3407Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 5m 539Mi ds-cts-2 5m 502Mi ds-idrepo-0 1332m 11793Mi ds-idrepo-1 2382m 9869Mi ds-idrepo-2 14m 5816Mi end-user-ui-58878d7668-zkqbp 1m 4Mi idm-6c95d7588f-98n7b 4172m 4589Mi idm-6c95d7588f-j6s6c 3443m 4515Mi lodemon-6b9df475cd-csxlf 1m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 260m 874Mi 10:30:42 DEBUG --- stderr --- 10:30:42 DEBUG 10:30:46 INFO 10:30:46 INFO [loop_until]: kubectl --namespace=xlou top node 10:30:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:30:46 INFO [loop_until]: OK (rc = 0) 10:30:46 DEBUG --- stdout --- 10:30:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 67m 0% 8246Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 2469m 15% 11144Mi 19% gke-xlou-cdm-default-pool-9d1e5395-jnd9 1481m 9% 13093Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 63m 0% 1754Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 376m 2% 6155Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 62m 0% 1738Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 58m 0% 1637Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 86m 0% 4968Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4500m 28% 6153Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 144m 0% 2429Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 71m 0% 4836Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 64m 0% 1193Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3241m 20% 5818Mi 9% gke-xlou-cdm-frontend-a8771548-i82n 342m 2% 4308Mi 7% 10:30:46 DEBUG --- stderr --- 10:30:46 DEBUG 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "instance": "10.142.0.24:9100" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "instance": "10.142.0.74:9100" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "12067.533333333333" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 ERROR [http_cmd]: ERROR 10:31:31 ERROR --- status code --- 10:31:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:31:31 ERROR --- http response --- 10:31:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:31:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-cts-1" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "ds-cts-0" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-idrepo-1" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-idrepo-1" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-idrepo-1" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-cts-1" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "ds-cts-0" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG ----- output ----- 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG { 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG "data": { 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG "result": [ 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG { 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "instance": "10.142.0.24:9100" 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG }, 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG "value": [ 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-idrepo-1" 10:31:31 DEBUG 10:31:31 DEBUG "instance": "10.142.0.24:9100" 10:31:31 DEBUG }, 10:31:31 DEBUG [...] 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "instance": "10.142.0.74:9100" 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG }, 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "9110.433333333334" 10:31:31 DEBUG 10:31:31 DEBUG "pod": "ds-idrepo-0" 10:31:31 DEBUG ] 10:31:31 DEBUG "instance": "10.142.0.74:9100" 10:31:31 DEBUG }, 10:31:31 DEBUG } 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG ] 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG } 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "771.8" 10:31:31 DEBUG } 10:31:31 DEBUG "0.00016666666666675193" 10:31:31 DEBUG ] 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951091" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "instance": "10.142.0.24:9100" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "instance": "10.142.0.74:9100" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "2885944320" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-idrepo-1" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "ds-idrepo-0" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "771.8" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0.2972871039968431" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:31:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:31:31 DEBUG }, 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:31:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:31:31 DEBUG }, 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-idrepo-1" 10:31:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:31:31 DEBUG }, 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "ds-idrepo-0" 10:31:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:31:31 DEBUG }, 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "771.7666666666667" 10:31:31 DEBUG "0.015" 10:31:31 DEBUG ] 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-idrepo-1" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-cts-1" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "ds-cts-0" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "1015902208" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "pod": "ds-idrepo-1" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "pod": "ds-cts-0" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951091" 10:31:31 INFO [http_cmd]: http status code OK 10:31:31 DEBUG --- status code --- 10:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:31:31 DEBUG --- http response --- 10:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:31:31 DEBUG ----- output ----- 10:31:31 DEBUG { 10:31:31 DEBUG "status": "success", 10:31:31 DEBUG "data": { 10:31:31 DEBUG "resultType": "vector", 10:31:31 DEBUG "result": [ 10:31:31 DEBUG { 10:31:31 DEBUG "metric": { 10:31:31 DEBUG "instance": "10.142.0.24:9100" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG [...] 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG 10:31:31 DEBUG "instance": "10.142.0.74:9100" 10:31:31 DEBUG }, 10:31:31 DEBUG "value": [ 10:31:31 DEBUG 1713951091, 10:31:31 DEBUG "0" 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG ] 10:31:31 DEBUG } 10:31:31 DEBUG } 10:31:42 INFO 10:31:42 INFO [loop_until]: kubectl --namespace=xlou top pods 10:31:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:31:42 INFO [loop_until]: OK (rc = 0) 10:31:42 DEBUG --- stdout --- 10:31:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3465Mi am-769b98f8d5-2rr6f 5m 3408Mi am-769b98f8d5-h2r87 3m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 6m 540Mi ds-cts-2 5m 502Mi ds-idrepo-0 3850m 11797Mi ds-idrepo-1 1023m 9856Mi ds-idrepo-2 9m 5817Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4085m 4694Mi idm-6c95d7588f-j6s6c 3368m 4512Mi lodemon-6b9df475cd-csxlf 4m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 270m 876Mi 10:31:42 DEBUG --- stderr --- 10:31:42 DEBUG 10:31:46 INFO 10:31:46 INFO [loop_until]: kubectl --namespace=xlou top node 10:31:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:31:46 INFO [loop_until]: OK (rc = 0) 10:31:46 DEBUG --- stdout --- 10:31:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 67m 0% 8233Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 626m 3% 11127Mi 18% gke-xlou-cdm-default-pool-9d1e5395-jnd9 3959m 24% 13103Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 58m 0% 1746Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 345m 2% 6153Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 61m 0% 1728Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1629Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 67m 0% 4974Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4443m 27% 6260Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 127m 0% 2437Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 68m 0% 4825Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 61m 0% 1207Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3752m 23% 5819Mi 9% gke-xlou-cdm-frontend-a8771548-i82n 341m 2% 4307Mi 7% 10:31:46 DEBUG --- stderr --- 10:31:46 DEBUG 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "instance": "10.142.0.24:9100" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "instance": "10.142.0.74:9100" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "13022.366666666663" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 ERROR [http_cmd]: ERROR 10:32:31 ERROR --- status code --- 10:32:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:32:31 ERROR --- http response --- 10:32:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:32:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "ds-cts-1" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "ds-cts-0" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "ds-idrepo-1" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "62024.70592532904" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:32:31 DEBUG "pod": "ds-idrepo-1" 10:32:31 DEBUG }, 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:32:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:32:31 DEBUG }, 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "instance": "10.142.0.24:9100" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "instance": "10.142.0.74:9100" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0.0002666666666666373" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "ds-idrepo-1" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0.09066344936248812" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "ds-idrepo-1" 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG "instance": "10.142.0.24:9100" 10:32:31 DEBUG }, 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG { 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG "data": { 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "result": [ 10:32:31 DEBUG 10:32:31 DEBUG "pod": "ds-idrepo-0" 10:32:31 DEBUG { 10:32:31 DEBUG "instance": "10.142.0.74:9100" 10:32:31 DEBUG }, 10:32:31 DEBUG "metric": { 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG "pod": "ds-cts-1" 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG }, 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "421.4" 10:32:31 DEBUG "value": [ 10:32:31 DEBUG "2886142976" 10:32:31 DEBUG ] 10:32:31 DEBUG 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG [...] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 DEBUG 10:32:31 DEBUG } 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "ds-cts-0" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG "result": [ 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG { 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "ds-idrepo-1" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:32:31 DEBUG 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG 10:32:31 DEBUG { 10:32:31 DEBUG 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG [...] 10:32:31 DEBUG "data": { 10:32:31 DEBUG 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG 10:32:31 DEBUG "result": [ 10:32:31 DEBUG 10:32:31 DEBUG { 10:32:31 DEBUG "pod": "ds-idrepo-0" 10:32:31 DEBUG "metric": { 10:32:31 DEBUG }, 10:32:31 DEBUG "instance": "10.142.0.24:9100" 10:32:31 DEBUG "value": [ 10:32:31 DEBUG }, 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG "421.4" 10:32:31 DEBUG 10:32:31 DEBUG ] 10:32:31 DEBUG 10:32:31 DEBUG } 10:32:31 DEBUG 10:32:31 DEBUG ] 10:32:31 DEBUG [...] 10:32:31 DEBUG } 10:32:31 DEBUG 10:32:31 DEBUG } 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "instance": "10.142.0.74:9100" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "7868.866666666667" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0.28821627900666535" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "1102942208" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "ds-idrepo-1" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "ds-idrepo-0" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG "421.4" 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG ] 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG } 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:32:31 DEBUG ] 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG } 10:32:31 DEBUG { 10:32:31 DEBUG } 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0.015" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "ds-idrepo-1" 10:32:31 DEBUG }, 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "ds-cts-1" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "ds-cts-0" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "pod": "ds-idrepo-1" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "pod": "ds-cts-0" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951151" 10:32:31 INFO [http_cmd]: http status code OK 10:32:31 DEBUG --- status code --- 10:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:32:31 DEBUG --- http response --- 10:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:32:31 DEBUG ----- output ----- 10:32:31 DEBUG { 10:32:31 DEBUG "status": "success", 10:32:31 DEBUG "data": { 10:32:31 DEBUG "resultType": "vector", 10:32:31 DEBUG "result": [ 10:32:31 DEBUG { 10:32:31 DEBUG "metric": { 10:32:31 DEBUG "instance": "10.142.0.24:9100" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG [...] 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG 10:32:31 DEBUG "instance": "10.142.0.74:9100" 10:32:31 DEBUG }, 10:32:31 DEBUG "value": [ 10:32:31 DEBUG 1713951151, 10:32:31 DEBUG "0" 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG ] 10:32:31 DEBUG } 10:32:31 DEBUG } 10:32:42 INFO 10:32:42 INFO [loop_until]: kubectl --namespace=xlou top pods 10:32:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:32:42 INFO [loop_until]: OK (rc = 0) 10:32:42 DEBUG --- stdout --- 10:32:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 7m 3466Mi am-769b98f8d5-2rr6f 4m 3407Mi am-769b98f8d5-h2r87 5m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 6m 540Mi ds-cts-2 10m 502Mi ds-idrepo-0 3720m 11799Mi ds-idrepo-1 950m 11553Mi ds-idrepo-2 8m 5817Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4361m 4699Mi idm-6c95d7588f-j6s6c 4161m 4718Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 294m 957Mi 10:32:42 DEBUG --- stderr --- 10:32:42 DEBUG 10:32:46 INFO 10:32:46 INFO [loop_until]: kubectl --namespace=xlou top node 10:32:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:32:46 INFO [loop_until]: OK (rc = 0) 10:32:46 DEBUG --- stdout --- 10:32:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 66m 0% 8220Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 924m 5% 12873Mi 21% gke-xlou-cdm-default-pool-9d1e5395-jnd9 3864m 24% 13108Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1750Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 352m 2% 6180Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1731Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 63m 0% 1640Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 85m 0% 4987Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4163m 26% 6272Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 155m 0% 2457Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 66m 0% 4823Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 61m 0% 1188Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 4613m 29% 6011Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 338m 2% 4386Mi 7% 10:32:46 DEBUG --- stderr --- 10:32:46 DEBUG 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "ds-cts-1" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "ds-cts-0" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 ERROR [http_cmd]: ERROR 10:33:31 ERROR --- status code --- 10:33:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:33:31 ERROR --- http response --- 10:33:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "instance": "10.142.0.24:9100" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "instance": "10.142.0.74:9100" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "11932.466666666667" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "ds-idrepo-1" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG --- http response --- 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951211" 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:33:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "instance": "10.142.0.24:9100" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "instance": "10.142.0.74:9100" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "2886378496" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "ds-idrepo-1" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "instance": "10.142.0.24:9100" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "instance": "10.142.0.74:9100" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0.00030000000000001136" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "ds-idrepo-1" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "ds-idrepo-1" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 DEBUG "pod": "ds-cts-1" 10:33:31 DEBUG }, 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "ds-idrepo-0" 10:33:31 DEBUG "pod": "ds-cts-0" 10:33:31 DEBUG }, 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "185.4" 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG { 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:33:31 DEBUG "result": [ 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "instance": "10.142.0.24:9100" 10:33:31 DEBUG "data": { 10:33:31 DEBUG }, 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "value": [ 10:33:31 DEBUG "result": [ 10:33:31 DEBUG 10:33:31 DEBUG { 10:33:31 DEBUG 10:33:31 DEBUG "metric": { 10:33:31 DEBUG 10:33:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:33:31 DEBUG [...] 10:33:31 DEBUG }, 10:33:31 DEBUG 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "instance": "10.142.0.74:9100" 10:33:31 DEBUG 10:33:31 DEBUG }, 10:33:31 DEBUG [...] 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG 10:33:31 DEBUG "6838.433333333333" 10:33:31 DEBUG 10:33:31 DEBUG ] 10:33:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:33:31 DEBUG } 10:33:31 DEBUG }, 10:33:31 DEBUG ] 10:33:31 DEBUG "value": [ 10:33:31 DEBUG } 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG } 10:33:31 DEBUG "0.2815592928370981" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "ds-idrepo-1" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "ds-idrepo-0" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "185.43333333333334" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "1102028800" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "ds-idrepo-1" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0.015" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG }, 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG "value": [ 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:33:31 DEBUG ] 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG } 10:33:31 DEBUG { 10:33:31 DEBUG ] 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG } 10:33:31 DEBUG "data": { 10:33:31 DEBUG } 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "ds-cts-1" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "ds-cts-0" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "ds-idrepo-1" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "ds-idrepo-0" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "185.43333333333334" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "pod": "ds-idrepo-1" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "pod": "ds-cts-0" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951211" 10:33:31 INFO [http_cmd]: http status code OK 10:33:31 DEBUG --- status code --- 10:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:33:31 DEBUG --- http response --- 10:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:33:31 DEBUG ----- output ----- 10:33:31 DEBUG { 10:33:31 DEBUG "status": "success", 10:33:31 DEBUG "data": { 10:33:31 DEBUG "resultType": "vector", 10:33:31 DEBUG "result": [ 10:33:31 DEBUG { 10:33:31 DEBUG "metric": { 10:33:31 DEBUG "instance": "10.142.0.24:9100" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG [...] 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG 10:33:31 DEBUG "instance": "10.142.0.74:9100" 10:33:31 DEBUG }, 10:33:31 DEBUG "value": [ 10:33:31 DEBUG 1713951211, 10:33:31 DEBUG "0" 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG ] 10:33:31 DEBUG } 10:33:31 DEBUG } 10:33:42 INFO 10:33:42 INFO [loop_until]: kubectl --namespace=xlou top pods 10:33:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:33:42 INFO [loop_until]: OK (rc = 0) 10:33:42 DEBUG --- stdout --- 10:33:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 5m 3466Mi am-769b98f8d5-2rr6f 5m 3407Mi am-769b98f8d5-h2r87 5m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 5m 540Mi ds-cts-2 6m 502Mi ds-idrepo-0 3109m 11802Mi ds-idrepo-1 179m 11773Mi ds-idrepo-2 17m 5818Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4108m 4698Mi idm-6c95d7588f-j6s6c 3003m 4717Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 248m 956Mi 10:33:42 DEBUG --- stderr --- 10:33:42 DEBUG 10:33:47 INFO 10:33:47 INFO [loop_until]: kubectl --namespace=xlou top node 10:33:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:33:47 INFO [loop_until]: OK (rc = 0) 10:33:47 DEBUG --- stdout --- 10:33:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 79m 0% 8234Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 77m 0% 13036Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4309m 27% 13103Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 59m 0% 1751Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 355m 2% 6153Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 58m 0% 1714Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 60m 0% 1644Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 87m 0% 4974Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4277m 26% 6262Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 149m 0% 2442Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 71m 0% 4814Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 64m 0% 1197Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 2504m 15% 6014Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 337m 2% 4386Mi 7% 10:33:47 DEBUG --- stderr --- 10:33:47 DEBUG 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 ERROR [http_cmd]: ERROR 10:34:31 ERROR --- status code --- 10:34:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:34:31 ERROR --- http response --- 10:34:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:34:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "ds-cts-1" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "ds-cts-0" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951271" 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "instance": "10.142.0.24:9100" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "instance": "10.142.0.74:9100" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "12106.833333333334" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG { 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG "data": { 10:34:31 DEBUG { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG "data": { 10:34:31 DEBUG { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "result": [ 10:34:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:34:31 DEBUG { 10:34:31 DEBUG }, 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "value": [ 10:34:31 DEBUG "instance": "10.142.0.24:9100" 10:34:31 DEBUG 10:34:31 DEBUG }, 10:34:31 DEBUG 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:34:31 DEBUG 10:34:31 DEBUG }, 10:34:31 DEBUG 10:34:31 DEBUG "value": [ 10:34:31 DEBUG "instance": "10.142.0.74:9100" 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG }, 10:34:31 DEBUG "0" 10:34:31 DEBUG "value": [ 10:34:31 DEBUG ] 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG } 10:34:31 DEBUG "2886620160" 10:34:31 DEBUG ] 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "133426801.7531612" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951271" 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG "result": [ 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG { 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG "metric": { 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:34:31 DEBUG }, 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG "value": [ 10:34:31 DEBUG { 10:34:31 DEBUG 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG 10:34:31 DEBUG "data": { 10:34:31 DEBUG 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG [...] 10:34:31 DEBUG "result": [ 10:34:31 DEBUG 10:34:31 DEBUG { 10:34:31 DEBUG 10:34:31 DEBUG "metric": { 10:34:31 DEBUG 10:34:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:34:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:34:31 DEBUG }, 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG 10:34:31 DEBUG "443.196692148661" 10:34:31 DEBUG 10:34:31 DEBUG ] 10:34:31 DEBUG [...] 10:34:31 DEBUG } 10:34:31 DEBUG 10:34:31 DEBUG ] 10:34:31 DEBUG 10:34:31 DEBUG } 10:34:31 DEBUG 10:34:31 DEBUG } 10:34:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0.29768611645267523" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "instance": "10.142.0.24:9100" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "instance": "10.142.0.74:9100" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0.00030000000000001136" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "1.2960654819341704" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "ds-cts-1" 10:34:31 DEBUG "pod": "ds-idrepo-1" 10:34:31 DEBUG }, 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "ds-cts-0" 10:34:31 DEBUG "pod": "ds-idrepo-0" 10:34:31 DEBUG }, 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0" 10:34:31 DEBUG "497.3" 10:34:31 DEBUG ] 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "instance": "10.142.0.24:9100" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "instance": "10.142.0.74:9100" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "6669.9333333333325" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "ds-idrepo-1" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "ds-idrepo-0" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "497.26666666666665" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951271" 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "1102778368" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "ds-idrepo-1" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "ds-idrepo-0" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "497.26666666666665" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0.015" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:34:31 DEBUG "pod": "ds-cts-1" 10:34:31 DEBUG }, 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:34:31 DEBUG "pod": "ds-cts-0" 10:34:31 DEBUG }, 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0" 10:34:31 DEBUG "0" 10:34:31 DEBUG ] 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951271" 10:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951271" 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "pod": "ds-idrepo-1" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "pod": "ds-cts-0" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:31 INFO [http_cmd]: http status code OK 10:34:31 DEBUG --- status code --- 10:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:34:31 DEBUG --- http response --- 10:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:34:31 DEBUG ----- output ----- 10:34:31 DEBUG { 10:34:31 DEBUG "status": "success", 10:34:31 DEBUG "data": { 10:34:31 DEBUG "resultType": "vector", 10:34:31 DEBUG "result": [ 10:34:31 DEBUG { 10:34:31 DEBUG "metric": { 10:34:31 DEBUG "instance": "10.142.0.24:9100" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG [...] 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG 10:34:31 DEBUG "instance": "10.142.0.74:9100" 10:34:31 DEBUG }, 10:34:31 DEBUG "value": [ 10:34:31 DEBUG 1713951271, 10:34:31 DEBUG "0" 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG ] 10:34:31 DEBUG } 10:34:31 DEBUG } 10:34:42 INFO 10:34:42 INFO [loop_until]: kubectl --namespace=xlou top pods 10:34:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:34:42 INFO [loop_until]: OK (rc = 0) 10:34:42 DEBUG --- stdout --- 10:34:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 10m 3408Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 6m 540Mi ds-cts-2 5m 502Mi ds-idrepo-0 11m 11797Mi ds-idrepo-1 2434m 11776Mi ds-idrepo-2 8m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4462m 4702Mi idm-6c95d7588f-j6s6c 2215m 4718Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 257m 959Mi 10:34:42 DEBUG --- stderr --- 10:34:42 DEBUG 10:34:47 INFO 10:34:47 INFO [loop_until]: kubectl --namespace=xlou top node 10:34:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:34:47 INFO [loop_until]: OK (rc = 0) 10:34:47 DEBUG --- stdout --- 10:34:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 66m 0% 8233Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 2537m 15% 13059Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 1712m 10% 13102Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1749Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 350m 2% 6180Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 58m 0% 1728Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 57m 0% 1647Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 82m 0% 4977Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4338m 27% 6262Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 128m 0% 2448Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 71m 0% 4829Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 57m 0% 1202Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3848m 24% 6029Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 330m 2% 4388Mi 7% 10:34:47 DEBUG --- stderr --- 10:34:47 DEBUG 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 ERROR [http_cmd]: ERROR 10:35:31 ERROR --- status code --- 10:35:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:35:31 ERROR --- http response --- 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:35:31 DEBUG --- status code --- 10:35:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "instance": "10.142.0.24:9100" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "instance": "10.142.0.74:9100" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "12653.733333333332" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "pod": "ds-cts-1" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "ds-cts-0" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "0" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "0" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "0" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951331" 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG { 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG { 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG "metric": { 10:35:31 DEBUG { 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG }, 10:35:31 DEBUG "data": { 10:35:31 DEBUG "value": [ 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG "resultType": "vector", 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG "result": [ 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG 10:35:31 DEBUG { 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG { 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG "metric": { 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [...] 10:35:31 DEBUG "data": { 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 DEBUG "instance": "10.142.0.24:9100" 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 DEBUG 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG }, 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "value": [ 10:35:31 DEBUG { 10:35:31 DEBUG 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "data": { 10:35:31 DEBUG 10:35:31 DEBUG "data": { 10:35:31 DEBUG }, 10:35:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "value": [ 10:35:31 DEBUG }, 10:35:31 DEBUG "result": [ 10:35:31 DEBUG [...] 10:35:31 DEBUG "result": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG { 10:35:31 DEBUG 10:35:31 DEBUG { 10:35:31 DEBUG "0" 10:35:31 DEBUG 10:35:31 DEBUG "metric": { 10:35:31 DEBUG 10:35:31 DEBUG "metric": { 10:35:31 DEBUG ] 10:35:31 DEBUG 10:35:31 DEBUG "instance": "10.142.0.24:9100" 10:35:31 DEBUG 10:35:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:35:31 DEBUG } 10:35:31 DEBUG 10:35:31 DEBUG }, 10:35:31 DEBUG "instance": "10.142.0.74:9100" 10:35:31 DEBUG }, 10:35:31 DEBUG ] 10:35:31 DEBUG [...] 10:35:31 DEBUG "value": [ 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG } 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG } 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "2887035904" 10:35:31 DEBUG 10:35:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:35:31 DEBUG [...] 10:35:31 DEBUG ] 10:35:31 DEBUG [...] 10:35:31 DEBUG }, 10:35:31 DEBUG 10:35:31 DEBUG } 10:35:31 DEBUG 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG ] 10:35:31 DEBUG 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG 10:35:31 DEBUG } 10:35:31 DEBUG 10:35:31 DEBUG "0.3377239698183305" 10:35:31 DEBUG "instance": "10.142.0.74:9100" 10:35:31 DEBUG } 10:35:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:35:31 DEBUG ] 10:35:31 DEBUG }, 10:35:31 DEBUG }, 10:35:31 DEBUG } 10:35:31 DEBUG "value": [ 10:35:31 DEBUG "value": [ 10:35:31 DEBUG ] 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG } 10:35:31 DEBUG "0.0002666666666667557" 10:35:31 DEBUG "7062.8246880148035" 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "0.09893558081155113" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG { 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "result": [ 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG { 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "pod": "ds-cts-1" 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "instance": "10.142.0.24:9100" 10:35:31 DEBUG }, 10:35:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:35:31 DEBUG "pod": "ds-idrepo-1" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG }, 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG "value": [ 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "ds-cts-0" 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "instance": "10.142.0.74:9100" 10:35:31 DEBUG }, 10:35:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:35:31 DEBUG "pod": "ds-idrepo-0" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG }, 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "0" 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "7070.5" 10:35:31 DEBUG ] 10:35:31 DEBUG "0" 10:35:31 DEBUG "511.6666666666667" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "0" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "data": { 10:35:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG }, 10:35:31 DEBUG "result": [ 10:35:31 DEBUG "value": [ 10:35:31 DEBUG { 10:35:31 DEBUG 10:35:31 DEBUG "metric": { 10:35:31 DEBUG 10:35:31 DEBUG "pod": "ds-idrepo-1" 10:35:31 DEBUG 10:35:31 DEBUG }, 10:35:31 DEBUG [...] 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:35:31 DEBUG [...] 10:35:31 DEBUG }, 10:35:31 DEBUG 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG 10:35:31 DEBUG "0" 10:35:31 DEBUG "pod": "ds-idrepo-0" 10:35:31 DEBUG ] 10:35:31 DEBUG }, 10:35:31 DEBUG } 10:35:31 DEBUG "value": [ 10:35:31 DEBUG ] 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG } 10:35:31 DEBUG "511.6666666666667" 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951331" 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "0" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "0" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "1103245312" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "pod": "ds-idrepo-1" 10:35:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:35:31 DEBUG }, 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "ds-idrepo-0" 10:35:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:35:31 DEBUG }, 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "511.6333333333333" 10:35:31 DEBUG "0.015" 10:35:31 DEBUG ] 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951331" 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG { 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG "data": { 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "data": { 10:35:31 DEBUG "pod": "ds-cts-1" 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG }, 10:35:31 DEBUG "result": [ 10:35:31 DEBUG "value": [ 10:35:31 DEBUG { 10:35:31 DEBUG 10:35:31 DEBUG "metric": { 10:35:31 DEBUG 10:35:31 DEBUG "pod": "ds-idrepo-1" 10:35:31 DEBUG 10:35:31 DEBUG }, 10:35:31 DEBUG [...] 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "pod": "ds-cts-0" 10:35:31 DEBUG [...] 10:35:31 DEBUG }, 10:35:31 DEBUG 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG 10:35:31 DEBUG "0" 10:35:31 DEBUG "pod": "ds-cts-0" 10:35:31 DEBUG ] 10:35:31 DEBUG }, 10:35:31 DEBUG } 10:35:31 DEBUG "value": [ 10:35:31 DEBUG ] 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG } 10:35:31 DEBUG "0" 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951331" 10:35:31 INFO [http_cmd]: http status code OK 10:35:31 DEBUG --- status code --- 10:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:35:31 DEBUG --- http response --- 10:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:35:31 DEBUG ----- output ----- 10:35:31 DEBUG { 10:35:31 DEBUG "status": "success", 10:35:31 DEBUG "data": { 10:35:31 DEBUG "resultType": "vector", 10:35:31 DEBUG "result": [ 10:35:31 DEBUG { 10:35:31 DEBUG "metric": { 10:35:31 DEBUG "instance": "10.142.0.24:9100" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG [...] 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG 10:35:31 DEBUG "instance": "10.142.0.74:9100" 10:35:31 DEBUG }, 10:35:31 DEBUG "value": [ 10:35:31 DEBUG 1713951331, 10:35:31 DEBUG "0" 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG ] 10:35:31 DEBUG } 10:35:31 DEBUG } 10:35:42 INFO 10:35:42 INFO [loop_until]: kubectl --namespace=xlou top pods 10:35:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:35:42 INFO [loop_until]: OK (rc = 0) 10:35:42 DEBUG --- stdout --- 10:35:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 5m 3408Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 6m 541Mi ds-cts-2 5m 502Mi ds-idrepo-0 1903m 11799Mi ds-idrepo-1 2514m 11782Mi ds-idrepo-2 10m 5818Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 2990m 4702Mi idm-6c95d7588f-j6s6c 4176m 4755Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 282m 960Mi 10:35:42 DEBUG --- stderr --- 10:35:42 DEBUG 10:35:47 INFO 10:35:47 INFO [loop_until]: kubectl --namespace=xlou top node 10:35:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:35:47 INFO [loop_until]: OK (rc = 0) 10:35:47 DEBUG --- stdout --- 10:35:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 67m 0% 8238Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 2750m 17% 13055Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 1791m 11% 13091Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1754Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 362m 2% 6190Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 57m 0% 1724Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1645Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 81m 0% 4976Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 2550m 16% 6255Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 151m 0% 2446Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 66m 0% 4827Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 60m 0% 1193Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 5804m 36% 6036Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 349m 2% 4384Mi 7% 10:35:47 DEBUG --- stderr --- 10:35:47 DEBUG 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951391" 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "instance": "10.142.0.24:9100" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "instance": "10.142.0.74:9100" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "12489.7" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 ERROR [http_cmd]: ERROR 10:36:31 ERROR --- status code --- 10:36:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:36:31 ERROR --- http response --- 10:36:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:36:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "pod": "ds-cts-1" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "ds-cts-0" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "0" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG "result": [ 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG { 10:36:31 DEBUG { 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG { 10:36:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:36:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG }, 10:36:31 DEBUG }, 10:36:31 DEBUG "data": { 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "result": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG { 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "metric": { 10:36:31 DEBUG [...] 10:36:31 DEBUG [...] 10:36:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG }, 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:36:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:36:31 DEBUG 10:36:31 DEBUG }, 10:36:31 DEBUG }, 10:36:31 DEBUG 10:36:31 DEBUG "value": [ 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 DEBUG "value": [ 10:36:31 DEBUG [...] 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG 10:36:31 DEBUG "0" 10:36:31 DEBUG "0" 10:36:31 DEBUG 10:36:31 DEBUG ] 10:36:31 DEBUG ] 10:36:31 DEBUG 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:36:31 DEBUG ] 10:36:31 DEBUG ] 10:36:31 DEBUG }, 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG "value": [ 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "0.3241741516505211" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "pod": "ds-idrepo-1" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "0" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "pod": "ds-idrepo-1" 10:36:31 DEBUG }, 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG "value": [ 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG [...] 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG { 10:36:31 DEBUG { 10:36:31 DEBUG 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "data": { 10:36:31 DEBUG "data": { 10:36:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG }, 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG "result": [ 10:36:31 DEBUG "result": [ 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG { 10:36:31 DEBUG { 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "0" 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "instance": "10.142.0.24:9100" 10:36:31 DEBUG "pod": "ds-idrepo-1" 10:36:31 DEBUG "instance": "10.142.0.24:9100" 10:36:31 DEBUG ] 10:36:31 DEBUG "pod": "ds-idrepo-1" 10:36:31 DEBUG }, 10:36:31 DEBUG }, 10:36:31 DEBUG }, 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "value": [ 10:36:31 DEBUG ] 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG } 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG } 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG [...] 10:36:31 DEBUG [...] 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG { 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "data": { 10:36:31 DEBUG 10:36:31 DEBUG "instance": "10.142.0.74:9100" 10:36:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:36:31 DEBUG "instance": "10.142.0.74:9100" 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:36:31 DEBUG }, 10:36:31 DEBUG }, 10:36:31 DEBUG }, 10:36:31 DEBUG "result": [ 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "value": [ 10:36:31 DEBUG { 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "metric": { 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "2887277568" 10:36:31 DEBUG "0.1402962389080167" 10:36:31 DEBUG "0.0002666666666666373" 10:36:31 DEBUG "pod": "ds-cts-1" 10:36:31 DEBUG "122042.61810674099" 10:36:31 DEBUG ] 10:36:31 DEBUG ] 10:36:31 DEBUG ] 10:36:31 DEBUG }, 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG "value": [ 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG ] 10:36:31 DEBUG ] 10:36:31 DEBUG 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG 10:36:31 DEBUG } 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "ds-cts-0" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "0" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG "data": { 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG "result": [ 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG { 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG "metric": { 10:36:31 DEBUG { 10:36:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG }, 10:36:31 DEBUG "data": { 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG 10:36:31 DEBUG "result": [ 10:36:31 DEBUG 10:36:31 DEBUG { 10:36:31 DEBUG 10:36:31 DEBUG "metric": { 10:36:31 DEBUG [...] 10:36:31 DEBUG "instance": "10.142.0.24:9100" 10:36:31 DEBUG 10:36:31 DEBUG }, 10:36:31 DEBUG 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:36:31 DEBUG 10:36:31 DEBUG }, 10:36:31 DEBUG 10:36:31 DEBUG "value": [ 10:36:31 DEBUG [...] 10:36:31 DEBUG 1713951391, 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 DEBUG 10:36:31 DEBUG "0" 10:36:31 DEBUG 10:36:31 DEBUG ] 10:36:31 DEBUG 10:36:31 DEBUG } 10:36:31 DEBUG "instance": "10.142.0.74:9100" 10:36:31 DEBUG ] 10:36:31 DEBUG }, 10:36:31 DEBUG } 10:36:31 DEBUG "value": [ 10:36:31 DEBUG } 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "9702.099999999999" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG "pod": "ds-idrepo-1" 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG }, 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG "value": [ 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:36:31 DEBUG 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG 10:36:31 DEBUG { 10:36:31 DEBUG [...] 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG 10:36:31 DEBUG "data": { 10:36:31 DEBUG 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG 10:36:31 DEBUG "result": [ 10:36:31 DEBUG "pod": "ds-idrepo-0" 10:36:31 DEBUG { 10:36:31 DEBUG }, 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG }, 10:36:31 DEBUG "503.3" 10:36:31 DEBUG "value": [ 10:36:31 DEBUG ] 10:36:31 DEBUG 10:36:31 DEBUG } 10:36:31 DEBUG 10:36:31 DEBUG ] 10:36:31 DEBUG 10:36:31 DEBUG } 10:36:31 DEBUG [...] 10:36:31 DEBUG } 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "0" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "pod": "ds-idrepo-1" 10:36:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:36:31 DEBUG }, 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "ds-idrepo-0" 10:36:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:36:31 DEBUG }, 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "503.3" 10:36:31 DEBUG "0" 10:36:31 DEBUG ] 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "pod": "ds-idrepo-1" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "ds-idrepo-0" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "503.26666666666665" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "0" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "1204879360" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "pod": "ds-cts-1" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "ds-cts-0" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "0" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "0.015" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951391" 10:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951391" 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "instance": "10.142.0.24:9100" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "instance": "10.142.0.74:9100" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "0.0003333333333333299" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:31 INFO [http_cmd]: http status code OK 10:36:31 DEBUG --- status code --- 10:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:36:31 DEBUG --- http response --- 10:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:36:31 DEBUG ----- output ----- 10:36:31 DEBUG { 10:36:31 DEBUG "status": "success", 10:36:31 DEBUG "data": { 10:36:31 DEBUG "resultType": "vector", 10:36:31 DEBUG "result": [ 10:36:31 DEBUG { 10:36:31 DEBUG "metric": { 10:36:31 DEBUG "pod": "ds-idrepo-1" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG [...] 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG 10:36:31 DEBUG "pod": "ds-cts-0" 10:36:31 DEBUG }, 10:36:31 DEBUG "value": [ 10:36:31 DEBUG 1713951391, 10:36:31 DEBUG "0" 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG ] 10:36:31 DEBUG } 10:36:31 DEBUG } 10:36:42 INFO 10:36:42 INFO [loop_until]: kubectl --namespace=xlou top pods 10:36:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:36:42 INFO [loop_until]: OK (rc = 0) 10:36:42 DEBUG --- stdout --- 10:36:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 5m 3466Mi am-769b98f8d5-2rr6f 5m 3409Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 6m 541Mi ds-cts-2 11m 502Mi ds-idrepo-0 1447m 11798Mi ds-idrepo-1 2727m 11781Mi ds-idrepo-2 8m 5818Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4519m 4702Mi idm-6c95d7588f-j6s6c 4688m 4733Mi lodemon-6b9df475cd-csxlf 1m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 218m 1058Mi 10:36:42 DEBUG --- stderr --- 10:36:42 DEBUG 10:36:47 INFO 10:36:47 INFO [loop_until]: kubectl --namespace=xlou top node 10:36:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:36:47 INFO [loop_until]: OK (rc = 0) 10:36:47 DEBUG --- stdout --- 10:36:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 67m 0% 8236Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 2504m 15% 13053Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 1847m 11% 13100Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1753Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 357m 2% 6194Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1720Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 61m 0% 1646Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 83m 0% 4973Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4564m 28% 6263Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 139m 0% 2450Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 66m 0% 4831Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 58m 0% 1207Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 4557m 28% 6043Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 340m 2% 4481Mi 7% 10:36:47 DEBUG --- stderr --- 10:36:47 DEBUG 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951451" 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "ds-cts-1" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "ds-cts-0" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "instance": "10.142.0.24:9100" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "instance": "10.142.0.74:9100" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "12787.533333333333" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951451" 10:37:31 ERROR [http_cmd]: ERROR 10:37:31 ERROR --- status code --- 10:37:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:37:31 ERROR --- http response --- 10:37:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:37:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "data": { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "ds-cts-1" 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "instance": "10.142.0.24:9100" 10:37:31 DEBUG }, 10:37:31 DEBUG "result": [ 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG { 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG "metric": { 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG }, 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG "value": [ 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "ds-cts-0" 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG [...] 10:37:31 DEBUG "instance": "10.142.0.74:9100" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 DEBUG }, 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:37:31 DEBUG 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0" 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:37:31 DEBUG "0.0002666666666667557" 10:37:31 DEBUG ] 10:37:31 DEBUG { 10:37:31 DEBUG }, 10:37:31 DEBUG ] 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG } 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "value": [ 10:37:31 DEBUG } 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG ] 10:37:31 DEBUG "data": { 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG ] 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG } 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "0.33087825367616547" 10:37:31 DEBUG } 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG } 10:37:31 DEBUG "result": [ 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:37:31 DEBUG { 10:37:31 DEBUG } 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG "metric": { 10:37:31 DEBUG ] 10:37:31 DEBUG { 10:37:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:37:31 DEBUG } 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG }, 10:37:31 DEBUG } 10:37:31 DEBUG "data": { 10:37:31 DEBUG "value": [ 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG 10:37:31 DEBUG "result": [ 10:37:31 DEBUG 10:37:31 DEBUG { 10:37:31 DEBUG 10:37:31 DEBUG "metric": { 10:37:31 DEBUG [...] 10:37:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:37:31 DEBUG 10:37:31 DEBUG }, 10:37:31 DEBUG 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:37:31 DEBUG 10:37:31 DEBUG }, 10:37:31 DEBUG 10:37:31 DEBUG "value": [ 10:37:31 DEBUG [...] 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG 10:37:31 DEBUG "0" 10:37:31 DEBUG 10:37:31 DEBUG ] 10:37:31 DEBUG 10:37:31 DEBUG } 10:37:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:37:31 DEBUG ] 10:37:31 DEBUG }, 10:37:31 DEBUG } 10:37:31 DEBUG "value": [ 10:37:31 DEBUG } 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "ds-idrepo-1" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0.06666666666666667" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "ds-idrepo-1" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "ds-idrepo-0" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "496.76666666666665" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "ds-idrepo-1" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG "data": { 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG "result": [ 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG { 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:37:31 DEBUG "metric": { 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG "pod": "ds-idrepo-1" 10:37:31 DEBUG { 10:37:31 DEBUG }, 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "value": [ 10:37:31 DEBUG "data": { 10:37:31 DEBUG 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG 10:37:31 DEBUG "result": [ 10:37:31 DEBUG 10:37:31 DEBUG { 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG "metric": { 10:37:31 DEBUG 10:37:31 DEBUG "instance": "10.142.0.24:9100" 10:37:31 DEBUG 10:37:31 DEBUG }, 10:37:31 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:37:31 DEBUG "value": [ 10:37:31 DEBUG }, 10:37:31 DEBUG 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG 10:37:31 DEBUG "0.00001832315271396018" 10:37:31 DEBUG [...] 10:37:31 DEBUG ] 10:37:31 DEBUG 10:37:31 DEBUG } 10:37:31 DEBUG 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG ] 10:37:31 DEBUG 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG } 10:37:31 DEBUG "instance": "10.142.0.74:9100" 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG } 10:37:31 DEBUG }, 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:37:31 DEBUG "2887535616" 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG ] 10:37:31 DEBUG { 10:37:31 DEBUG } 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG ] 10:37:31 DEBUG "data": { 10:37:31 DEBUG } 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG } 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "ds-idrepo-1" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "instance": "10.142.0.24:9100" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "instance": "10.142.0.74:9100" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "7586.066666666667" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "ds-cts-1" 10:37:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:37:31 DEBUG }, 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "ds-cts-0" 10:37:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:37:31 DEBUG }, 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0" 10:37:31 DEBUG "0" 10:37:31 DEBUG ] 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "ds-idrepo-1" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "ds-idrepo-0" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "496.76666666666665" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "ds-idrepo-1" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "ds-idrepo-0" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "496.73333333333335" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951451" 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0.015" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951451" 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "instance": "10.142.0.24:9100" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "instance": "10.142.0.74:9100" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0.06666666666666667" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "1207631872" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951451" 10:37:31 INFO [http_cmd]: http status code OK 10:37:31 DEBUG --- status code --- 10:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:37:31 DEBUG --- http response --- 10:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:37:31 DEBUG ----- output ----- 10:37:31 DEBUG { 10:37:31 DEBUG "status": "success", 10:37:31 DEBUG "data": { 10:37:31 DEBUG "resultType": "vector", 10:37:31 DEBUG "result": [ 10:37:31 DEBUG { 10:37:31 DEBUG "metric": { 10:37:31 DEBUG "pod": "ds-idrepo-1" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG [...] 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG 10:37:31 DEBUG "pod": "ds-cts-0" 10:37:31 DEBUG }, 10:37:31 DEBUG "value": [ 10:37:31 DEBUG 1713951451, 10:37:31 DEBUG "0" 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG ] 10:37:31 DEBUG } 10:37:31 DEBUG } 10:37:42 INFO 10:37:42 INFO [loop_until]: kubectl --namespace=xlou top pods 10:37:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:37:42 INFO [loop_until]: OK (rc = 0) 10:37:42 DEBUG --- stdout --- 10:37:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 6m 3408Mi am-769b98f8d5-h2r87 5m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 6m 541Mi ds-cts-2 5m 502Mi ds-idrepo-0 11m 11798Mi ds-idrepo-1 3162m 11782Mi ds-idrepo-2 9m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 5347m 4704Mi idm-6c95d7588f-j6s6c 4348m 4734Mi lodemon-6b9df475cd-csxlf 4m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 295m 1058Mi 10:37:42 DEBUG --- stderr --- 10:37:42 DEBUG 10:37:47 INFO 10:37:47 INFO [loop_until]: kubectl --namespace=xlou top node 10:37:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:37:47 INFO [loop_until]: OK (rc = 0) 10:37:47 DEBUG --- stdout --- 10:37:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 65m 0% 8248Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 2951m 18% 13058Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 1833m 11% 13102Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 64m 0% 1755Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 389m 2% 6200Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1725Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 57m 0% 1643Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 88m 0% 4978Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4294m 27% 6268Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 131m 0% 2439Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 69m 0% 4832Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 62m 0% 1202Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3755m 23% 6045Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 358m 2% 4481Mi 7% 10:37:47 DEBUG --- stderr --- 10:37:47 DEBUG 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951511" 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "instance": "10.142.0.24:9100" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "instance": "10.142.0.74:9100" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "13695.366666666667" 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "pod": "ds-cts-1" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "pod": "ds-cts-0" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "0" 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951511" 10:38:31 ERROR [http_cmd]: ERROR 10:38:31 ERROR --- status code --- 10:38:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:38:31 ERROR --- http response --- 10:38:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:38:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "pod": "ds-idrepo-1" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "547.9415404167084" 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951511" 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 DEBUG --- status code --- 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG { 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "status": "success", 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG "data": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "data": { 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG "result": [ 10:38:31 DEBUG "result": [ 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG { 10:38:31 DEBUG { 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "pod": "ds-cts-1" 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG "pod": "ds-idrepo-1" 10:38:31 DEBUG }, 10:38:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:38:31 DEBUG "pod": "ds-idrepo-1" 10:38:31 DEBUG { 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG }, 10:38:31 DEBUG }, 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG "value": [ 10:38:31 DEBUG "value": [ 10:38:31 DEBUG "data": { 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "result": [ 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG { 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG [...] 10:38:31 DEBUG "metric": { 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "instance": "10.142.0.24:9100" 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG }, 10:38:31 DEBUG 10:38:31 DEBUG "pod": "ds-cts-0" 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "value": [ 10:38:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:38:31 DEBUG }, 10:38:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:38:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:38:31 DEBUG 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG }, 10:38:31 DEBUG }, 10:38:31 DEBUG 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "0" 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG [...] 10:38:31 DEBUG "0" 10:38:31 DEBUG ] 10:38:31 DEBUG "0" 10:38:31 DEBUG "136.9853851041771" 10:38:31 DEBUG 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG ] 10:38:31 DEBUG 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 DEBUG 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG ] 10:38:31 DEBUG ] 10:38:31 DEBUG "instance": "10.142.0.74:9100" 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 DEBUG }, 10:38:31 DEBUG } 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 DEBUG "value": [ 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:38:31 DEBUG "0.00020000000000000757" 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG ] 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG { 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG ] 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG "data": { 10:38:31 DEBUG } 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG } 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG "data": { 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:38:31 DEBUG { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG }, 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "data": { 10:38:31 DEBUG "value": [ 10:38:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG 10:38:31 DEBUG }, 10:38:31 DEBUG "result": [ 10:38:31 DEBUG 10:38:31 DEBUG "value": [ 10:38:31 DEBUG { 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "metric": { 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG }, 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG }, 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "value": [ 10:38:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:38:31 DEBUG [...] 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG }, 10:38:31 DEBUG 10:38:31 DEBUG "0" 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG ] 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG 10:38:31 DEBUG } 10:38:31 DEBUG "0.3567234844699887" 10:38:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:38:31 DEBUG ] 10:38:31 DEBUG ] 10:38:31 DEBUG }, 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 DEBUG "value": [ 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG } 10:38:31 DEBUG "0" 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "pod": "ds-idrepo-1" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG "pod": "ds-idrepo-0" 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG }, 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:38:31 DEBUG "value": [ 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG "724.8333333333334" 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG ] 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG "status": "success", 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG "data": { 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG } 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG "data": { 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:38:31 DEBUG ] 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG "result": [ 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG } 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG "result": [ 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG { 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG { 10:38:31 DEBUG } 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG "data": { 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG "instance": "10.142.0.24:9100" 10:38:31 DEBUG }, 10:38:31 DEBUG { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG "data": { 10:38:31 DEBUG { 10:38:31 DEBUG "data": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "result": [ 10:38:31 DEBUG "pod": "ds-idrepo-1" 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG { 10:38:31 DEBUG }, 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "value": [ 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:38:31 DEBUG 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 DEBUG 10:38:31 DEBUG "pod": "ds-idrepo-1" 10:38:31 DEBUG 10:38:31 DEBUG "pod": "ds-idrepo-1" 10:38:31 DEBUG }, 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG }, 10:38:31 DEBUG 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG "instance": "10.142.0.74:9100" 10:38:31 DEBUG }, 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "0" 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG "2887818240" 10:38:31 DEBUG ] 10:38:31 DEBUG { 10:38:31 DEBUG 10:38:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG 10:38:31 DEBUG }, 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG "data": { 10:38:31 DEBUG 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "pod": "ds-idrepo-0" 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "pod": "ds-idrepo-0" 10:38:31 DEBUG }, 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 DEBUG "result": [ 10:38:31 DEBUG }, 10:38:31 DEBUG "0.0036047623825291586" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG } 10:38:31 DEBUG { 10:38:31 DEBUG "value": [ 10:38:31 DEBUG ] 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "metric": { 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG } 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "0" 10:38:31 DEBUG "instance": "10.142.0.24:9100" 10:38:31 DEBUG "724.8" 10:38:31 DEBUG ] 10:38:31 DEBUG "724.8" 10:38:31 DEBUG ] 10:38:31 DEBUG }, 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG "value": [ 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG 10:38:31 DEBUG ] 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 DEBUG 10:38:31 DEBUG } 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG } 10:38:31 DEBUG [...] 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:38:31 DEBUG 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG "instance": "10.142.0.74:9100" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 DEBUG { 10:38:31 DEBUG }, 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "value": [ 10:38:31 DEBUG "data": { 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "6849.966666666666" 10:38:31 DEBUG "result": [ 10:38:31 DEBUG ] 10:38:31 DEBUG { 10:38:31 DEBUG } 10:38:31 DEBUG "metric": { 10:38:31 DEBUG ] 10:38:31 DEBUG "pod": "ds-cts-1" 10:38:31 DEBUG } 10:38:31 DEBUG }, 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG "value": [ 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:38:31 DEBUG [...] 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG 10:38:31 DEBUG { 10:38:31 DEBUG 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG 10:38:31 DEBUG "data": { 10:38:31 DEBUG "pod": "ds-cts-0" 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG }, 10:38:31 DEBUG "result": [ 10:38:31 DEBUG "value": [ 10:38:31 DEBUG { 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "0" 10:38:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:38:31 DEBUG ] 10:38:31 DEBUG }, 10:38:31 DEBUG } 10:38:31 DEBUG "value": [ 10:38:31 DEBUG ] 10:38:31 DEBUG 10:38:31 DEBUG } 10:38:31 DEBUG 10:38:31 DEBUG } 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "0.015" 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "0" 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951511" 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951511" 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "instance": "10.142.0.24:9100" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "instance": "10.142.0.74:9100" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "0" 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951511" 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "1204248576" 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:31 INFO [http_cmd]: http status code OK 10:38:31 DEBUG --- status code --- 10:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:38:31 DEBUG --- http response --- 10:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:38:31 DEBUG ----- output ----- 10:38:31 DEBUG { 10:38:31 DEBUG "status": "success", 10:38:31 DEBUG "data": { 10:38:31 DEBUG "resultType": "vector", 10:38:31 DEBUG "result": [ 10:38:31 DEBUG { 10:38:31 DEBUG "metric": { 10:38:31 DEBUG "pod": "ds-idrepo-1" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG [...] 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG 10:38:31 DEBUG "pod": "ds-cts-0" 10:38:31 DEBUG }, 10:38:31 DEBUG "value": [ 10:38:31 DEBUG 1713951511, 10:38:31 DEBUG "0" 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG ] 10:38:31 DEBUG } 10:38:31 DEBUG } 10:38:42 INFO 10:38:42 INFO [loop_until]: kubectl --namespace=xlou top pods 10:38:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:38:42 INFO [loop_until]: OK (rc = 0) 10:38:42 DEBUG --- stdout --- 10:38:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 5m 3408Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 5m 541Mi ds-cts-2 5m 502Mi ds-idrepo-0 4275m 11792Mi ds-idrepo-1 20m 11781Mi ds-idrepo-2 18m 5818Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 9454m 4707Mi idm-6c95d7588f-j6s6c 621m 1004Mi lodemon-6b9df475cd-csxlf 1m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 287m 1056Mi 10:38:42 DEBUG --- stderr --- 10:38:42 DEBUG 10:38:47 INFO 10:38:47 INFO [loop_until]: kubectl --namespace=xlou top node 10:38:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:38:47 INFO [loop_until]: OK (rc = 0) 10:38:47 DEBUG --- stdout --- 10:38:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 78m 0% 8235Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 63m 0% 13051Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4475m 28% 13103Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1755Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 354m 2% 6167Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 57m 0% 1726Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1643Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 82m 0% 4980Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 9781m 61% 6271Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 149m 0% 2467Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 66m 0% 4836Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 60m 0% 1194Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 1329m 8% 2521Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 350m 2% 4483Mi 7% 10:38:47 DEBUG --- stderr --- 10:38:47 DEBUG 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951571" 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "data": { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "result": [ 10:39:31 DEBUG { 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "instance": "10.142.0.24:9100" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "instance": "10.142.0.74:9100" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "13165.233333333334" 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "data": { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "result": [ 10:39:31 DEBUG { 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "pod": "ds-cts-1" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "pod": "ds-cts-0" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "0" 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951571" 10:39:31 ERROR [http_cmd]: ERROR 10:39:31 ERROR --- status code --- 10:39:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:39:31 ERROR --- http response --- 10:39:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:39:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 DEBUG { 10:39:31 DEBUG { 10:39:31 DEBUG "status": "success", 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "data": { 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG "data": { 10:39:31 DEBUG "data": { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "result": [ 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG "result": [ 10:39:31 DEBUG "result": [ 10:39:31 DEBUG { 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:39:31 DEBUG { 10:39:31 DEBUG { 10:39:31 DEBUG "metric": { 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:39:31 DEBUG { 10:39:31 DEBUG "pod": "ds-cts-1" 10:39:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:39:31 DEBUG }, 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG }, 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG "data": { 10:39:31 DEBUG "value": [ 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "result": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG { 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG "metric": { 10:39:31 DEBUG [...] 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG }, 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:39:31 DEBUG 10:39:31 DEBUG "pod": "ds-cts-0" 10:39:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:39:31 DEBUG }, 10:39:31 DEBUG 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 DEBUG }, 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG "value": [ 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951571" 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG [...] 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "0" 10:39:31 DEBUG 10:39:31 DEBUG "0" 10:39:31 DEBUG "0" 10:39:31 DEBUG ] 10:39:31 DEBUG 10:39:31 DEBUG ] 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:39:31 DEBUG ] 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG }, 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG "value": [ 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "0" 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951571" 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG { 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "data": { 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "data": { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "data": { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "result": [ 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "result": [ 10:39:31 DEBUG { 10:39:31 DEBUG "result": [ 10:39:31 DEBUG { 10:39:31 DEBUG "metric": { 10:39:31 DEBUG { 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "instance": "10.142.0.24:9100" 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:39:31 DEBUG }, 10:39:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "instance": "10.142.0.74:9100" 10:39:31 DEBUG 10:39:31 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:39:31 DEBUG }, 10:39:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "0.00030000000000001136" 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "0" 10:39:31 DEBUG ] 10:39:31 DEBUG "0" 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG { 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG "data": { 10:39:31 DEBUG { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "result": [ 10:39:31 DEBUG "data": { 10:39:31 DEBUG { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "result": [ 10:39:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:39:31 DEBUG { 10:39:31 DEBUG }, 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "value": [ 10:39:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:39:31 DEBUG 10:39:31 DEBUG }, 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:39:31 DEBUG 10:39:31 DEBUG }, 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG }, 10:39:31 DEBUG "0" 10:39:31 DEBUG "value": [ 10:39:31 DEBUG ] 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG } 10:39:31 DEBUG "0.30685096167493253" 10:39:31 DEBUG ] 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951571" 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- http response --- 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG "data": { 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG "result": [ 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:39:31 DEBUG { 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG "metric": { 10:39:31 DEBUG { 10:39:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG }, 10:39:31 DEBUG "data": { 10:39:31 DEBUG "value": [ 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG 10:39:31 DEBUG "result": [ 10:39:31 DEBUG 10:39:31 DEBUG { 10:39:31 DEBUG 10:39:31 DEBUG "metric": { 10:39:31 DEBUG [...] 10:39:31 DEBUG "pod": "ds-idrepo-1" 10:39:31 DEBUG 10:39:31 DEBUG }, 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:39:31 DEBUG 10:39:31 DEBUG }, 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG [...] 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG 10:39:31 DEBUG "0" 10:39:31 DEBUG 10:39:31 DEBUG ] 10:39:31 DEBUG 10:39:31 DEBUG } 10:39:31 DEBUG "pod": "ds-idrepo-0" 10:39:31 DEBUG ] 10:39:31 DEBUG }, 10:39:31 DEBUG } 10:39:31 DEBUG "value": [ 10:39:31 DEBUG } 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "661.3" 10:39:31 DEBUG ] 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG } 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG ] 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG } 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG } 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "data": { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "result": [ 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG { 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG "metric": { 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG "pod": "ds-idrepo-1" 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG }, 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:39:31 DEBUG "value": [ 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG { 10:39:31 DEBUG 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG "data": { 10:39:31 DEBUG [...] 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951571" 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG "result": [ 10:39:31 DEBUG 10:39:31 DEBUG { 10:39:31 DEBUG { 10:39:31 DEBUG 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "pod": "ds-idrepo-0" 10:39:31 DEBUG "data": { 10:39:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:39:31 DEBUG }, 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG "result": [ 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG { 10:39:31 DEBUG 10:39:31 DEBUG "661.3666666666667" 10:39:31 DEBUG "metric": { 10:39:31 DEBUG 10:39:31 DEBUG ] 10:39:31 DEBUG "pod": "ds-idrepo-1" 10:39:31 DEBUG 10:39:31 DEBUG } 10:39:31 DEBUG }, 10:39:31 DEBUG [...] 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 DEBUG ] 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG } 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG } 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:39:31 DEBUG [...] 10:39:31 DEBUG }, 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG 10:39:31 DEBUG "1208233984" 10:39:31 DEBUG "pod": "ds-idrepo-0" 10:39:31 DEBUG ] 10:39:31 DEBUG }, 10:39:31 DEBUG } 10:39:31 DEBUG "value": [ 10:39:31 DEBUG ] 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG } 10:39:31 DEBUG "661.3666666666667" 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951571" 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "data": { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "result": [ 10:39:31 DEBUG { 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "instance": "10.142.0.24:9100" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "instance": "10.142.0.74:9100" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "2888086528" 10:39:31 DEBUG ] 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG } 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG ] 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG } 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG } 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG ----- output ----- 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG { 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG { 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG "data": { 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG "data": { 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:39:31 DEBUG "result": [ 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG "result": [ 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 DEBUG { 10:39:31 DEBUG { 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG { 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "metric": { 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:39:31 DEBUG "data": { 10:39:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG "data": { 10:39:31 DEBUG }, 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG }, 10:39:31 DEBUG { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "value": [ 10:39:31 DEBUG "result": [ 10:39:31 DEBUG "value": [ 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "result": [ 10:39:31 DEBUG 10:39:31 DEBUG { 10:39:31 DEBUG 10:39:31 DEBUG "data": { 10:39:31 DEBUG { 10:39:31 DEBUG 10:39:31 DEBUG "metric": { 10:39:31 DEBUG 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "metric": { 10:39:31 DEBUG 10:39:31 DEBUG "instance": "10.142.0.24:9100" 10:39:31 DEBUG 10:39:31 DEBUG "result": [ 10:39:31 DEBUG "pod": "ds-cts-1" 10:39:31 DEBUG [...] 10:39:31 DEBUG }, 10:39:31 DEBUG [...] 10:39:31 DEBUG { 10:39:31 DEBUG }, 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG }, 10:39:31 DEBUG 10:39:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:39:31 DEBUG 10:39:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG }, 10:39:31 DEBUG [...] 10:39:31 DEBUG }, 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 1713951571, 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "0" 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG 10:39:31 DEBUG "0" 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG ] 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG "instance": "10.142.0.74:9100" 10:39:31 DEBUG ] 10:39:31 DEBUG 10:39:31 DEBUG "pod": "ds-cts-0" 10:39:31 DEBUG } 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG }, 10:39:31 DEBUG } 10:39:31 DEBUG 10:39:31 DEBUG }, 10:39:31 DEBUG ] 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:39:31 DEBUG "value": [ 10:39:31 DEBUG ] 10:39:31 DEBUG 10:39:31 DEBUG "value": [ 10:39:31 DEBUG } 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG } 10:39:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG } 10:39:31 DEBUG { 10:39:31 DEBUG "6644.066666666667" 10:39:31 DEBUG } 10:39:31 DEBUG }, 10:39:31 DEBUG "0" 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG ] 10:39:31 DEBUG "value": [ 10:39:31 DEBUG ] 10:39:31 DEBUG "data": { 10:39:31 DEBUG } 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG } 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG ] 10:39:31 DEBUG "0.015" 10:39:31 DEBUG ] 10:39:31 DEBUG "result": [ 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG { 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 DEBUG "metric": { 10:39:31 DEBUG ] 10:39:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:39:31 DEBUG } 10:39:31 DEBUG }, 10:39:31 DEBUG } 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "0" 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951571" 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "data": { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "result": [ 10:39:31 DEBUG { 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "instance": "10.142.0.24:9100" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "instance": "10.142.0.74:9100" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "0" 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951571" 10:39:31 INFO [http_cmd]: http status code OK 10:39:31 DEBUG --- status code --- 10:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:39:31 DEBUG --- http response --- 10:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:39:31 DEBUG ----- output ----- 10:39:31 DEBUG { 10:39:31 DEBUG "status": "success", 10:39:31 DEBUG "data": { 10:39:31 DEBUG "resultType": "vector", 10:39:31 DEBUG "result": [ 10:39:31 DEBUG { 10:39:31 DEBUG "metric": { 10:39:31 DEBUG "pod": "ds-idrepo-1" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG [...] 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG 10:39:31 DEBUG "pod": "ds-cts-0" 10:39:31 DEBUG }, 10:39:31 DEBUG "value": [ 10:39:31 DEBUG 1713951571, 10:39:31 DEBUG "0" 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG ] 10:39:31 DEBUG } 10:39:31 DEBUG } 10:39:42 INFO 10:39:42 INFO [loop_until]: kubectl --namespace=xlou top pods 10:39:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:39:43 INFO [loop_until]: OK (rc = 0) 10:39:43 DEBUG --- stdout --- 10:39:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 5m 3466Mi am-769b98f8d5-2rr6f 4m 3408Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 6m 541Mi ds-cts-2 5m 501Mi ds-idrepo-0 2387m 11798Mi ds-idrepo-1 1950m 11792Mi ds-idrepo-2 9m 5818Mi end-user-ui-58878d7668-zkqbp 1m 4Mi idm-6c95d7588f-98n7b 3811m 4719Mi idm-6c95d7588f-j6s6c 2988m 2967Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 257m 1167Mi 10:39:43 DEBUG --- stderr --- 10:39:43 DEBUG 10:39:47 INFO 10:39:47 INFO [loop_until]: kubectl --namespace=xlou top node 10:39:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:39:47 INFO [loop_until]: OK (rc = 0) 10:39:47 DEBUG --- stdout --- 10:39:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 66m 0% 8236Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 1831m 11% 13067Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 2551m 16% 13102Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 59m 0% 1755Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 329m 2% 6137Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 57m 0% 1725Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 54m 0% 1639Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 84m 0% 4980Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 5496m 34% 6283Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 142m 0% 2438Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 68m 0% 4831Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 61m 0% 1194Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3569m 22% 4508Mi 7% gke-xlou-cdm-frontend-a8771548-i82n 327m 2% 4595Mi 7% 10:39:47 DEBUG --- stderr --- 10:39:47 DEBUG 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 ERROR [http_cmd]: ERROR 10:40:31 ERROR --- status code --- 10:40:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:40:31 ERROR --- http response --- 10:40:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:40:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- http response --- 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG { 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG "data": { 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG { 10:40:31 DEBUG "result": [ 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG { 10:40:31 DEBUG "data": { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "pod": "ds-cts-1" 10:40:31 DEBUG "result": [ 10:40:31 DEBUG }, 10:40:31 DEBUG { 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "metric": { 10:40:31 DEBUG 10:40:31 DEBUG "instance": "10.142.0.24:9100" 10:40:31 DEBUG 10:40:31 DEBUG }, 10:40:31 DEBUG 10:40:31 DEBUG "value": [ 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG "pod": "ds-cts-0" 10:40:31 DEBUG 10:40:31 DEBUG }, 10:40:31 DEBUG 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "instance": "10.142.0.74:9100" 10:40:31 DEBUG "0" 10:40:31 DEBUG }, 10:40:31 DEBUG ] 10:40:31 DEBUG "value": [ 10:40:31 DEBUG } 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG ] 10:40:31 DEBUG "11765.599999999999" 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- http response --- 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:40:31 DEBUG --- status code --- 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG { 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG { 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG { 10:40:31 DEBUG { 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "data": { 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "data": { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "data": { 10:40:31 DEBUG "data": { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "result": [ 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "result": [ 10:40:31 DEBUG { 10:40:31 DEBUG "result": [ 10:40:31 DEBUG "result": [ 10:40:31 DEBUG { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG { 10:40:31 DEBUG { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:40:31 DEBUG }, 10:40:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:40:31 DEBUG "instance": "10.142.0.24:9100" 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG }, 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG [...] 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG [...] 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG { 10:40:31 DEBUG 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:40:31 DEBUG 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG 10:40:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:40:31 DEBUG "data": { 10:40:31 DEBUG 10:40:31 DEBUG { 10:40:31 DEBUG 10:40:31 DEBUG { 10:40:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:40:31 DEBUG }, 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "instance": "10.142.0.74:9100" 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "result": [ 10:40:31 DEBUG }, 10:40:31 DEBUG "data": { 10:40:31 DEBUG }, 10:40:31 DEBUG "data": { 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG { 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "0" 10:40:31 DEBUG "metric": { 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "result": [ 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "result": [ 10:40:31 DEBUG "0" 10:40:31 DEBUG ] 10:40:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:40:31 DEBUG "507.5116934609547" 10:40:31 DEBUG { 10:40:31 DEBUG "0.00023333333333338166" 10:40:31 DEBUG { 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG }, 10:40:31 DEBUG ] 10:40:31 DEBUG "metric": { 10:40:31 DEBUG ] 10:40:31 DEBUG "metric": { 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG "value": [ 10:40:31 DEBUG } 10:40:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:40:31 DEBUG } 10:40:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG 10:40:31 DEBUG ] 10:40:31 DEBUG }, 10:40:31 DEBUG ] 10:40:31 DEBUG }, 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG 10:40:31 DEBUG } 10:40:31 DEBUG "value": [ 10:40:31 DEBUG } 10:40:31 DEBUG "value": [ 10:40:31 DEBUG } 10:40:31 DEBUG 10:40:31 DEBUG } 10:40:31 DEBUG 10:40:31 DEBUG } 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG }, 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "value": [ 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:40:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG }, 10:40:31 DEBUG }, 10:40:31 DEBUG "0.2834763192053041" 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "value": [ 10:40:31 DEBUG ] 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG } 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG "0" 10:40:31 DEBUG "0" 10:40:31 DEBUG ] 10:40:31 DEBUG { 10:40:31 DEBUG { 10:40:31 DEBUG ] 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG "data": { 10:40:31 DEBUG "data": { 10:40:31 DEBUG ] 10:40:31 DEBUG ] 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG "result": [ 10:40:31 DEBUG "result": [ 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG { 10:40:31 DEBUG { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "pod": "ds-cts-1" 10:40:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:40:31 DEBUG }, 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "pod": "ds-cts-0" 10:40:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:40:31 DEBUG }, 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "0" 10:40:31 DEBUG "112610627.76074095" 10:40:31 DEBUG ] 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG { 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "data": { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "result": [ 10:40:31 DEBUG { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "pod": "ds-cts-1" 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "pod": "ds-cts-0" 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "0" 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG { 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "data": { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "result": [ 10:40:31 DEBUG { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "pod": "ds-idrepo-1" 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "pod": "ds-idrepo-0" 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "678.7" 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG --- status code --- 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951631" 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- http response --- 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG { 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG { 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "data": { 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG "data": { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "result": [ 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "result": [ 10:40:31 DEBUG { 10:40:31 DEBUG "data": { 10:40:31 DEBUG { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "pod": "ds-idrepo-1" 10:40:31 DEBUG "result": [ 10:40:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:40:31 DEBUG }, 10:40:31 DEBUG { 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG }, 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG "pod": "ds-idrepo-0" 10:40:31 DEBUG 10:40:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:40:31 DEBUG }, 10:40:31 DEBUG 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:40:31 DEBUG 1713951631, 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951631" 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG "678.6666666666666" 10:40:31 DEBUG }, 10:40:31 DEBUG "0" 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG ] 10:40:31 DEBUG "value": [ 10:40:31 DEBUG ] 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG } 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG ] 10:40:31 DEBUG "0" 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG } 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG } 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG { 10:40:31 DEBUG { 10:40:31 DEBUG ] 10:40:31 DEBUG { 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG } 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "data": { 10:40:31 DEBUG "data": { 10:40:31 DEBUG } 10:40:31 DEBUG "data": { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "resultType": "vector", 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "result": [ 10:40:31 DEBUG "result": [ 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG "result": [ 10:40:31 DEBUG { 10:40:31 DEBUG { 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "pod": "ds-idrepo-1" 10:40:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:40:31 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:40:31 DEBUG }, 10:40:31 DEBUG }, 10:40:31 DEBUG }, 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "value": [ 10:40:31 DEBUG { 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "data": { 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG [...] 10:40:31 DEBUG [...] 10:40:31 DEBUG [...] 10:40:31 DEBUG "result": [ 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG { 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "metric": { 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:40:31 DEBUG "pod": "ds-idrepo-0" 10:40:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:40:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:40:31 DEBUG }, 10:40:31 DEBUG }, 10:40:31 DEBUG }, 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG 1713951631, 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG 10:40:31 DEBUG "678.6666666666666" 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG "1324851200" 10:40:31 DEBUG "0.003942074776198054" 10:40:31 DEBUG 10:40:31 DEBUG ] 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG ] 10:40:31 DEBUG ] 10:40:31 DEBUG [...] 10:40:31 DEBUG } 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG 10:40:31 DEBUG ] 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG ] 10:40:31 DEBUG ] 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951631" 10:40:31 DEBUG 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 DEBUG } 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG 10:40:31 DEBUG } 10:40:31 DEBUG { 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG }, 10:40:31 DEBUG "data": { 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "result": [ 10:40:31 DEBUG "0" 10:40:31 DEBUG { 10:40:31 DEBUG ] 10:40:31 DEBUG "metric": { 10:40:31 DEBUG } 10:40:31 DEBUG "instance": "10.142.0.24:9100" 10:40:31 DEBUG ] 10:40:31 DEBUG }, 10:40:31 DEBUG } 10:40:31 DEBUG "value": [ 10:40:31 DEBUG } 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "instance": "10.142.0.74:9100" 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "2888479744" 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:40:31 DEBUG ----- output ----- 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG { 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG "data": { 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "result": [ 10:40:31 DEBUG { 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG "metric": { 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:40:31 DEBUG { 10:40:31 DEBUG }, 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "data": { 10:40:31 DEBUG 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG 10:40:31 DEBUG "result": [ 10:40:31 DEBUG 10:40:31 DEBUG { 10:40:31 DEBUG [...] 10:40:31 DEBUG "metric": { 10:40:31 DEBUG 10:40:31 DEBUG "instance": "10.142.0.24:9100" 10:40:31 DEBUG 10:40:31 DEBUG }, 10:40:31 DEBUG 10:40:31 DEBUG "value": [ 10:40:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:40:31 DEBUG 10:40:31 DEBUG }, 10:40:31 DEBUG 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG [...] 10:40:31 DEBUG "0.015" 10:40:31 DEBUG 10:40:31 DEBUG ] 10:40:31 DEBUG 10:40:31 DEBUG } 10:40:31 DEBUG 10:40:31 DEBUG ] 10:40:31 DEBUG "instance": "10.142.0.74:9100" 10:40:31 DEBUG } 10:40:31 DEBUG }, 10:40:31 DEBUG } 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "6509.833333333333" 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951631" 10:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951631" 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG { 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "data": { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "result": [ 10:40:31 DEBUG { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "instance": "10.142.0.24:9100" 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "instance": "10.142.0.74:9100" 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "0.00033333333333333365" 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:31 INFO [http_cmd]: http status code OK 10:40:31 DEBUG --- status code --- 10:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:40:31 DEBUG --- http response --- 10:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:40:31 DEBUG ----- output ----- 10:40:31 DEBUG { 10:40:31 DEBUG "status": "success", 10:40:31 DEBUG "data": { 10:40:31 DEBUG "resultType": "vector", 10:40:31 DEBUG "result": [ 10:40:31 DEBUG { 10:40:31 DEBUG "metric": { 10:40:31 DEBUG "pod": "ds-idrepo-1" 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG [...] 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG 10:40:31 DEBUG "pod": "ds-cts-0" 10:40:31 DEBUG }, 10:40:31 DEBUG "value": [ 10:40:31 DEBUG 1713951631, 10:40:31 DEBUG "0" 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG ] 10:40:31 DEBUG } 10:40:31 DEBUG } 10:40:43 INFO 10:40:43 INFO [loop_until]: kubectl --namespace=xlou top pods 10:40:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:40:43 INFO [loop_until]: OK (rc = 0) 10:40:43 DEBUG --- stdout --- 10:40:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 4m 3408Mi am-769b98f8d5-h2r87 10m 3435Mi ds-cts-0 6m 563Mi ds-cts-1 7m 540Mi ds-cts-2 5m 502Mi ds-idrepo-0 3033m 11796Mi ds-idrepo-1 2155m 11804Mi ds-idrepo-2 15m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 3931m 4708Mi idm-6c95d7588f-j6s6c 3498m 2997Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 297m 1168Mi 10:40:43 DEBUG --- stderr --- 10:40:43 DEBUG 10:40:47 INFO 10:40:47 INFO [loop_until]: kubectl --namespace=xlou top node 10:40:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:40:47 INFO [loop_until]: OK (rc = 0) 10:40:47 DEBUG --- stdout --- 10:40:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 73m 0% 8238Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 1719m 10% 13072Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 2751m 17% 13090Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 59m 0% 1744Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 354m 2% 6136Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 58m 0% 1723Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 58m 0% 1638Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 84m 0% 4964Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4427m 27% 6269Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 144m 0% 2452Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 71m 0% 4840Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 61m 0% 1193Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3699m 23% 4556Mi 7% gke-xlou-cdm-frontend-a8771548-i82n 344m 2% 4597Mi 7% 10:40:47 DEBUG --- stderr --- 10:40:47 DEBUG 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951691" 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "ds-cts-1" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "ds-cts-0" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "0" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 ERROR [http_cmd]: ERROR 10:41:31 ERROR --- status code --- 10:41:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:41:31 ERROR --- http response --- 10:41:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:41:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951691" 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "instance": "10.142.0.24:9100" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "instance": "10.142.0.74:9100" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "12939.1" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "instance": "10.142.0.24:9100" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "instance": "10.142.0.74:9100" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "0.0001666666666666335" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "ds-idrepo-1" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "6016.57748399958" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "0" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951691" 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "ds-cts-1" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "ds-cts-0" 10:41:31 DEBUG }, 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 DEBUG "value": [ 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG --- status code --- 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG "0" 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG ] 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG } 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG ] 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:41:31 DEBUG } 10:41:31 DEBUG { 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG } 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG { 10:41:31 DEBUG "data": { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "result": [ 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG { 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG "result": [ 10:41:31 DEBUG "metric": { 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG { 10:41:31 DEBUG "pod": "ds-idrepo-1" 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG "metric": { 10:41:31 DEBUG }, 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:41:31 DEBUG "value": [ 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG }, 10:41:31 DEBUG 10:41:31 DEBUG { 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "data": { 10:41:31 DEBUG 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG [...] 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG "result": [ 10:41:31 DEBUG [...] 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG { 10:41:31 DEBUG 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG "metric": { 10:41:31 DEBUG 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:41:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:41:31 DEBUG 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG }, 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG }, 10:41:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:41:31 DEBUG { 10:41:31 DEBUG "value": [ 10:41:31 DEBUG { 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:41:31 DEBUG "value": [ 10:41:31 DEBUG }, 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG 10:41:31 DEBUG "value": [ 10:41:31 DEBUG "data": { 10:41:31 DEBUG "429.7555345713986" 10:41:31 DEBUG "data": { 10:41:31 DEBUG { 10:41:31 DEBUG 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG ] 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG 10:41:31 DEBUG "0" 10:41:31 DEBUG "result": [ 10:41:31 DEBUG } 10:41:31 DEBUG "result": [ 10:41:31 DEBUG "data": { 10:41:31 DEBUG [...] 10:41:31 DEBUG ] 10:41:31 DEBUG { 10:41:31 DEBUG ] 10:41:31 DEBUG { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG 10:41:31 DEBUG } 10:41:31 DEBUG "metric": { 10:41:31 DEBUG } 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "result": [ 10:41:31 DEBUG 10:41:31 DEBUG ] 10:41:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:41:31 DEBUG } 10:41:31 DEBUG "pod": "ds-idrepo-1" 10:41:31 DEBUG { 10:41:31 DEBUG 10:41:31 DEBUG } 10:41:31 DEBUG }, 10:41:31 DEBUG }, 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:41:31 DEBUG } 10:41:31 DEBUG "value": [ 10:41:31 DEBUG "value": [ 10:41:31 DEBUG "pod": "ds-cts-1" 10:41:31 DEBUG }, 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "0.2779272029670918" 10:41:31 DEBUG [...] 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG ] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG } 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG ] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG } 10:41:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:41:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:41:31 DEBUG 10:41:31 DEBUG } 10:41:31 DEBUG }, 10:41:31 DEBUG }, 10:41:31 DEBUG 10:41:31 DEBUG "value": [ 10:41:31 DEBUG "value": [ 10:41:31 DEBUG "pod": "ds-cts-0" 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG }, 10:41:31 DEBUG "0" 10:41:31 DEBUG "0" 10:41:31 DEBUG "value": [ 10:41:31 DEBUG ] 10:41:31 DEBUG ] 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG "0" 10:41:31 DEBUG ] 10:41:31 DEBUG ] 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "ds-idrepo-1" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "ds-idrepo-0" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "788.7666666666667" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "0.015" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "ds-idrepo-1" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "0.0035265274717585374" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "0" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951691" 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "0" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "status": "success", 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 DEBUG "data": { 10:41:31 DEBUG "data": { 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "ds-idrepo-1" 10:41:31 DEBUG "instance": "10.142.0.24:9100" 10:41:31 DEBUG }, 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "ds-idrepo-0" 10:41:31 DEBUG "instance": "10.142.0.74:9100" 10:41:31 DEBUG }, 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "788.7333333333333" 10:41:31 DEBUG "2888754176" 10:41:31 DEBUG ] 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951691" 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "pod": "ds-idrepo-1" 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:41:31 DEBUG }, 10:41:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "pod": "ds-idrepo-0" 10:41:31 DEBUG 10:41:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:41:31 DEBUG }, 10:41:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "788.7666666666667" 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "1321877504" 10:41:31 DEBUG ] 10:41:31 DEBUG "0" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: http status code OK 10:41:31 DEBUG --- status code --- 10:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:31 DEBUG --- http response --- 10:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:31 DEBUG ----- output ----- 10:41:31 DEBUG { 10:41:31 DEBUG "status": "success", 10:41:31 DEBUG "data": { 10:41:31 DEBUG "resultType": "vector", 10:41:31 DEBUG "result": [ 10:41:31 DEBUG { 10:41:31 DEBUG "metric": { 10:41:31 DEBUG "instance": "10.142.0.24:9100" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG [...] 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG 10:41:31 DEBUG "instance": "10.142.0.74:9100" 10:41:31 DEBUG }, 10:41:31 DEBUG "value": [ 10:41:31 DEBUG 1713951691, 10:41:31 DEBUG "8568.933333333332" 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG ] 10:41:31 DEBUG } 10:41:31 DEBUG } 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951691" 10:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951691" 10:41:32 INFO [http_cmd]: http status code OK 10:41:32 DEBUG --- status code --- 10:41:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:32 DEBUG --- http response --- 10:41:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:41:32 DEBUG ----- output ----- 10:41:32 DEBUG { 10:41:32 DEBUG "status": "success", 10:41:32 DEBUG "data": { 10:41:32 DEBUG "resultType": "vector", 10:41:32 DEBUG "result": [ 10:41:32 DEBUG { 10:41:32 DEBUG "metric": { 10:41:32 DEBUG "instance": "10.142.0.24:9100" 10:41:32 DEBUG }, 10:41:32 DEBUG "value": [ 10:41:32 DEBUG 10:41:32 DEBUG 10:41:32 DEBUG 10:41:32 DEBUG [...] 10:41:32 DEBUG 10:41:32 DEBUG 10:41:32 DEBUG 10:41:32 DEBUG "instance": "10.142.0.74:9100" 10:41:32 DEBUG }, 10:41:32 DEBUG "value": [ 10:41:32 DEBUG 1713951691, 10:41:32 DEBUG "0.00033333333333333365" 10:41:32 DEBUG ] 10:41:32 DEBUG } 10:41:32 DEBUG ] 10:41:32 DEBUG } 10:41:32 DEBUG } 10:41:32 INFO [http_cmd]: http status code OK 10:41:32 DEBUG --- status code --- 10:41:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:41:32 DEBUG --- http response --- 10:41:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:41:32 DEBUG ----- output ----- 10:41:32 DEBUG { 10:41:32 DEBUG "status": "success", 10:41:32 DEBUG "data": { 10:41:32 DEBUG "resultType": "vector", 10:41:32 DEBUG "result": [ 10:41:32 DEBUG { 10:41:32 DEBUG "metric": { 10:41:32 DEBUG "pod": "ds-idrepo-1" 10:41:32 DEBUG }, 10:41:32 DEBUG "value": [ 10:41:32 DEBUG 10:41:32 DEBUG 10:41:32 DEBUG 10:41:32 DEBUG [...] 10:41:32 DEBUG 10:41:32 DEBUG 10:41:32 DEBUG 10:41:32 DEBUG "pod": "ds-cts-0" 10:41:32 DEBUG }, 10:41:32 DEBUG "value": [ 10:41:32 DEBUG 1713951691, 10:41:32 DEBUG "0" 10:41:32 DEBUG ] 10:41:32 DEBUG } 10:41:32 DEBUG ] 10:41:32 DEBUG } 10:41:32 DEBUG } 10:41:43 INFO 10:41:43 INFO [loop_until]: kubectl --namespace=xlou top pods 10:41:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:41:43 INFO [loop_until]: OK (rc = 0) 10:41:43 DEBUG --- stdout --- 10:41:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 4m 3408Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 6m 562Mi ds-cts-1 6m 541Mi ds-cts-2 5m 502Mi ds-idrepo-0 4191m 11790Mi ds-idrepo-1 637m 11808Mi ds-idrepo-2 9m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4624m 4708Mi idm-6c95d7588f-j6s6c 3522m 3114Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 266m 1168Mi 10:41:43 DEBUG --- stderr --- 10:41:43 DEBUG 10:41:48 INFO 10:41:48 INFO [loop_until]: kubectl --namespace=xlou top node 10:41:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:41:48 INFO [loop_until]: OK (rc = 0) 10:41:48 DEBUG --- stdout --- 10:41:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 69m 0% 8245Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 542m 3% 13064Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4234m 26% 13100Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 59m 0% 1741Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 346m 2% 6143Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 61m 0% 1725Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 55m 0% 1632Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 83m 0% 4965Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4452m 28% 6272Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 155m 0% 2454Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 67m 0% 4822Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 60m 0% 1195Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3752m 23% 4668Mi 7% gke-xlou-cdm-frontend-a8771548-i82n 348m 2% 4597Mi 7% 10:41:48 DEBUG --- stderr --- 10:41:48 DEBUG 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951751" 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG { 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG "data": { 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG "result": [ 10:42:31 DEBUG { 10:42:31 DEBUG "metric": { 10:42:31 DEBUG "pod": "ds-cts-1" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG [...] 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG "pod": "ds-cts-0" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG "0" 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:31 ERROR [http_cmd]: ERROR 10:42:31 ERROR --- status code --- 10:42:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:42:31 ERROR --- http response --- 10:42:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:42:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951751" 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG { 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG "data": { 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG "result": [ 10:42:31 DEBUG { 10:42:31 DEBUG "metric": { 10:42:31 DEBUG "instance": "10.142.0.24:9100" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG [...] 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG "instance": "10.142.0.74:9100" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG "13038.9" 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951751" 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG { 10:42:31 DEBUG { 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG "data": { 10:42:31 DEBUG "data": { 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG "result": [ 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG { 10:42:31 DEBUG "result": [ 10:42:31 DEBUG "metric": { 10:42:31 DEBUG { 10:42:31 DEBUG "pod": "ds-cts-1" 10:42:31 DEBUG "metric": { 10:42:31 DEBUG }, 10:42:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:42:31 DEBUG "value": [ 10:42:31 DEBUG }, 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG 10:42:31 DEBUG "value": [ 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG [...] 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:42:31 DEBUG [...] 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG 10:42:31 DEBUG { 10:42:31 DEBUG 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG "pod": "ds-cts-0" 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG }, 10:42:31 DEBUG "data": { 10:42:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:42:31 DEBUG "value": [ 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG }, 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG "result": [ 10:42:31 DEBUG "value": [ 10:42:31 DEBUG { 10:42:31 DEBUG "0" 10:42:31 DEBUG { 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG ] 10:42:31 DEBUG "metric": { 10:42:31 DEBUG "0" 10:42:31 DEBUG "data": { 10:42:31 DEBUG } 10:42:31 DEBUG "instance": "10.142.0.24:9100" 10:42:31 DEBUG ] 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG ] 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG }, 10:42:31 DEBUG } 10:42:31 DEBUG "result": [ 10:42:31 DEBUG } 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG "value": [ 10:42:31 DEBUG ] 10:42:31 DEBUG { 10:42:31 DEBUG } 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG 10:42:31 DEBUG } 10:42:31 DEBUG "metric": { 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG 10:42:31 DEBUG } 10:42:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:42:31 DEBUG 10:42:31 DEBUG }, 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG [...] 10:42:31 DEBUG "value": [ 10:42:31 DEBUG { 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG "data": { 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG "instance": "10.142.0.74:9100" 10:42:31 DEBUG [...] 10:42:31 DEBUG "result": [ 10:42:31 DEBUG }, 10:42:31 DEBUG 10:42:31 DEBUG { 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 10:42:31 DEBUG "metric": { 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG 10:42:31 DEBUG "pod": "ds-idrepo-1" 10:42:31 DEBUG "0.00030000000000001136" 10:42:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:42:31 DEBUG }, 10:42:31 DEBUG ] 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG } 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 10:42:31 DEBUG ] 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG 10:42:31 DEBUG } 10:42:31 DEBUG "0" 10:42:31 DEBUG 10:42:31 DEBUG } 10:42:31 DEBUG ] 10:42:31 DEBUG [...] 10:42:31 DEBUG } 10:42:31 DEBUG 10:42:31 DEBUG ] 10:42:31 DEBUG 10:42:31 DEBUG } 10:42:31 DEBUG 10:42:31 DEBUG } 10:42:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG "124271793.24776977" 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG { 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG "data": { 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG "result": [ 10:42:31 DEBUG { 10:42:31 DEBUG "metric": { 10:42:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG [...] 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG "0" 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG { 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG "data": { 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG "result": [ 10:42:31 DEBUG { 10:42:31 DEBUG "metric": { 10:42:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG [...] 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG "0" 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951751" 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG { 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG "data": { 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG "result": [ 10:42:31 DEBUG { 10:42:31 DEBUG "metric": { 10:42:31 DEBUG "pod": "ds-idrepo-1" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG [...] 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:42:31 DEBUG }, 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG "411.95741007194243" 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG --- http response --- 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 INFO [http_cmd]: http status code OK 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG --- status code --- 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:31 DEBUG { 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG --- http response --- 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:42:31 DEBUG "data": { 10:42:31 DEBUG ----- output ----- 10:42:31 DEBUG ----- output ----- 10:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951751" 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG { 10:42:31 DEBUG { 10:42:31 DEBUG "result": [ 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG "status": "success", 10:42:31 DEBUG { 10:42:31 DEBUG "data": { 10:42:31 DEBUG "data": { 10:42:31 DEBUG "metric": { 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG "resultType": "vector", 10:42:31 DEBUG "pod": "ds-cts-1" 10:42:31 DEBUG "result": [ 10:42:31 DEBUG "result": [ 10:42:31 DEBUG }, 10:42:31 DEBUG { 10:42:31 DEBUG { 10:42:31 DEBUG "value": [ 10:42:31 DEBUG "metric": { 10:42:31 DEBUG "metric": { 10:42:31 DEBUG 10:42:31 DEBUG "pod": "ds-idrepo-1" 10:42:31 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:42:31 DEBUG 10:42:31 DEBUG }, 10:42:31 DEBUG }, 10:42:31 DEBUG 10:42:31 DEBUG "value": [ 10:42:31 DEBUG "value": [ 10:42:31 DEBUG [...] 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG [...] 10:42:31 DEBUG [...] 10:42:31 DEBUG "pod": "ds-cts-0" 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG }, 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG "value": [ 10:42:31 DEBUG 10:42:31 DEBUG 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG "pod": "idm-6c95d7588f-98n7b" 10:42:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:42:31 DEBUG "0" 10:42:31 DEBUG }, 10:42:31 DEBUG }, 10:42:31 DEBUG ] 10:42:31 DEBUG "value": [ 10:42:31 DEBUG "value": [ 10:42:31 DEBUG } 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG 1713951751, 10:42:31 DEBUG ] 10:42:31 DEBUG "0" 10:42:31 DEBUG "0.30666466224044814" 10:42:31 DEBUG } 10:42:31 DEBUG ] 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:31 DEBUG ] 10:42:31 DEBUG ] 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:31 DEBUG } 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG { 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG "data": { 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG "result": [ 10:42:32 DEBUG { 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "pod": "ds-idrepo-1" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG [...] 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG "pod": "ds-idrepo-0" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 1713951751, 10:42:32 DEBUG "544.1666666666666" 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951752" 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951752" 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951752" 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951752" 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG { 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG "data": { 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG "result": [ 10:42:32 DEBUG { 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG [...] 10:42:32 DEBUG 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG }, 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:42:32 DEBUG "value": [ 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG { 10:42:32 DEBUG "0" 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG ] 10:42:32 DEBUG "data": { 10:42:32 DEBUG } 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG ] 10:42:32 DEBUG "result": [ 10:42:32 DEBUG } 10:42:32 DEBUG { 10:42:32 DEBUG } 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG [...] 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG "0.015" 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951752" 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG --- http response --- 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG { 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:42:32 DEBUG "data": { 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG { 10:42:32 DEBUG "result": [ 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG { 10:42:32 DEBUG "data": { 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG "pod": "ds-idrepo-1" 10:42:32 DEBUG "result": [ 10:42:32 DEBUG }, 10:42:32 DEBUG { 10:42:32 DEBUG "value": [ 10:42:32 DEBUG "metric": { 10:42:32 DEBUG 10:42:32 DEBUG "pod": "ds-idrepo-1" 10:42:32 DEBUG 10:42:32 DEBUG }, 10:42:32 DEBUG 10:42:32 DEBUG "value": [ 10:42:32 DEBUG [...] 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG [...] 10:42:32 DEBUG "pod": "ds-idrepo-0" 10:42:32 DEBUG 10:42:32 DEBUG }, 10:42:32 DEBUG 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:42:32 DEBUG "544.2666666666667" 10:42:32 DEBUG }, 10:42:32 DEBUG ] 10:42:32 DEBUG "value": [ 10:42:32 DEBUG } 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG ] 10:42:32 DEBUG "0.004922405723455912" 10:42:32 DEBUG } 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG { 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG "data": { 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG "result": [ 10:42:32 DEBUG { 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG [...] 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG "1322901504" 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951752" 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951752" 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951752" 10:42:32 DEBUG --- status code --- 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG { 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951752" 10:42:32 DEBUG { 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG "data": { 10:42:32 DEBUG "data": { 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG "result": [ 10:42:32 DEBUG "result": [ 10:42:32 DEBUG { 10:42:32 DEBUG { 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "pod": "ds-idrepo-1" 10:42:32 DEBUG "instance": "10.142.0.24:9100" 10:42:32 DEBUG }, 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG [...] 10:42:32 DEBUG [...] 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG "pod": "ds-idrepo-0" 10:42:32 DEBUG "instance": "10.142.0.74:9100" 10:42:32 DEBUG }, 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG "544.2666666666667" 10:42:32 DEBUG "2888981504" 10:42:32 DEBUG ] 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 DEBUG ] 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG { 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG "data": { 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG "result": [ 10:42:32 DEBUG { 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "instance": "10.142.0.24:9100" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG [...] 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG "instance": "10.142.0.74:9100" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG "7598" 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG { 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG "data": { 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG "result": [ 10:42:32 DEBUG { 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG [...] 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG "0" 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951752" 10:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951752" 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG { 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG "data": { 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG "result": [ 10:42:32 DEBUG { 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "instance": "10.142.0.24:9100" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG [...] 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG "instance": "10.142.0.74:9100" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG "0" 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:32 INFO [http_cmd]: http status code OK 10:42:32 DEBUG --- status code --- 10:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:42:32 DEBUG --- http response --- 10:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:42:32 DEBUG ----- output ----- 10:42:32 DEBUG { 10:42:32 DEBUG "status": "success", 10:42:32 DEBUG "data": { 10:42:32 DEBUG "resultType": "vector", 10:42:32 DEBUG "result": [ 10:42:32 DEBUG { 10:42:32 DEBUG "metric": { 10:42:32 DEBUG "pod": "ds-idrepo-1" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG [...] 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG 10:42:32 DEBUG "pod": "ds-cts-0" 10:42:32 DEBUG }, 10:42:32 DEBUG "value": [ 10:42:32 DEBUG 1713951752, 10:42:32 DEBUG "0" 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG ] 10:42:32 DEBUG } 10:42:32 DEBUG } 10:42:43 INFO 10:42:43 INFO [loop_until]: kubectl --namespace=xlou top pods 10:42:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:42:43 INFO [loop_until]: OK (rc = 0) 10:42:43 DEBUG --- stdout --- 10:42:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 3m 3466Mi am-769b98f8d5-2rr6f 4m 3408Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 563Mi ds-cts-1 6m 541Mi ds-cts-2 5m 501Mi ds-idrepo-0 4055m 11794Mi ds-idrepo-1 11m 11816Mi ds-idrepo-2 10m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4050m 4500Mi idm-6c95d7588f-j6s6c 3299m 3149Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 266m 1169Mi 10:42:43 DEBUG --- stderr --- 10:42:43 DEBUG 10:42:48 INFO 10:42:48 INFO [loop_until]: kubectl --namespace=xlou top node 10:42:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:42:48 INFO [loop_until]: OK (rc = 0) 10:42:48 DEBUG --- stdout --- 10:42:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 64m 0% 8235Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 51m 0% 13085Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4549m 28% 13099Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 60m 0% 1763Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 342m 2% 6149Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 57m 0% 1728Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1645Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 81m 0% 4974Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4120m 25% 6061Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 146m 0% 2439Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 65m 0% 4826Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 59m 0% 1191Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3315m 20% 4706Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 333m 2% 4597Mi 7% 10:42:48 DEBUG --- stderr --- 10:42:48 DEBUG 10:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951811" 10:43:31 INFO [http_cmd]: http status code OK 10:43:31 DEBUG --- status code --- 10:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:31 DEBUG --- http response --- 10:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:43:31 DEBUG ----- output ----- 10:43:31 DEBUG { 10:43:31 DEBUG "status": "success", 10:43:31 DEBUG "data": { 10:43:31 DEBUG "resultType": "vector", 10:43:31 DEBUG "result": [ 10:43:31 DEBUG { 10:43:31 DEBUG "metric": { 10:43:31 DEBUG "pod": "ds-cts-1" 10:43:31 DEBUG }, 10:43:31 DEBUG "value": [ 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG [...] 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG "pod": "ds-cts-0" 10:43:31 DEBUG }, 10:43:31 DEBUG "value": [ 10:43:31 DEBUG 1713951811, 10:43:31 DEBUG "0" 10:43:31 DEBUG ] 10:43:31 DEBUG } 10:43:31 DEBUG ] 10:43:31 DEBUG } 10:43:31 DEBUG } 10:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951811" 10:43:31 ERROR [http_cmd]: ERROR 10:43:31 ERROR --- status code --- 10:43:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:43:31 ERROR --- http response --- 10:43:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:43:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951811" 10:43:31 INFO [http_cmd]: http status code OK 10:43:31 DEBUG --- status code --- 10:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:31 DEBUG --- http response --- 10:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:43:31 DEBUG ----- output ----- 10:43:31 DEBUG { 10:43:31 DEBUG "status": "success", 10:43:31 DEBUG "data": { 10:43:31 DEBUG "resultType": "vector", 10:43:31 DEBUG "result": [ 10:43:31 DEBUG { 10:43:31 DEBUG "metric": { 10:43:31 DEBUG "instance": "10.142.0.24:9100" 10:43:31 DEBUG }, 10:43:31 DEBUG "value": [ 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG [...] 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG "instance": "10.142.0.74:9100" 10:43:31 DEBUG }, 10:43:31 DEBUG "value": [ 10:43:31 DEBUG 1713951811, 10:43:31 DEBUG "11906.333333333332" 10:43:31 DEBUG ] 10:43:31 DEBUG } 10:43:31 DEBUG ] 10:43:31 DEBUG } 10:43:31 DEBUG } 10:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951811" 10:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951811" 10:43:31 INFO [http_cmd]: http status code OK 10:43:31 DEBUG --- status code --- 10:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:31 DEBUG --- http response --- 10:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:43:31 DEBUG ----- output ----- 10:43:31 DEBUG { 10:43:31 DEBUG "status": "success", 10:43:31 DEBUG "data": { 10:43:31 DEBUG "resultType": "vector", 10:43:31 DEBUG "result": [ 10:43:31 DEBUG { 10:43:31 DEBUG "metric": { 10:43:31 DEBUG "pod": "am-769b98f8d5-h2r87" 10:43:31 DEBUG }, 10:43:31 DEBUG "value": [ 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG [...] 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:43:31 DEBUG }, 10:43:31 DEBUG "value": [ 10:43:31 DEBUG 1713951811, 10:43:31 DEBUG "0" 10:43:31 DEBUG ] 10:43:31 DEBUG } 10:43:31 DEBUG ] 10:43:31 DEBUG } 10:43:31 DEBUG } 10:43:31 INFO [http_cmd]: http status code OK 10:43:31 DEBUG --- status code --- 10:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:31 DEBUG --- http response --- 10:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:43:31 DEBUG ----- output ----- 10:43:31 DEBUG { 10:43:31 DEBUG "status": "success", 10:43:31 DEBUG "data": { 10:43:31 DEBUG "resultType": "vector", 10:43:31 DEBUG "result": [ 10:43:31 DEBUG { 10:43:31 DEBUG "metric": { 10:43:31 DEBUG "pod": "ds-cts-1" 10:43:31 DEBUG }, 10:43:31 DEBUG "value": [ 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG [...] 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG 10:43:31 DEBUG "pod": "ds-cts-0" 10:43:31 DEBUG }, 10:43:31 DEBUG "value": [ 10:43:31 DEBUG 1713951811, 10:43:31 DEBUG "0" 10:43:31 DEBUG ] 10:43:31 DEBUG } 10:43:31 DEBUG ] 10:43:31 DEBUG } 10:43:31 DEBUG } 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG { 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:43:32 DEBUG "data": { 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:43:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG }, 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG { 10:43:32 DEBUG { 10:43:32 DEBUG "value": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:43:32 DEBUG "data": { 10:43:32 DEBUG "data": { 10:43:32 DEBUG 10:43:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:43:32 DEBUG }, 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG "result": [ 10:43:32 DEBUG "result": [ 10:43:32 DEBUG [...] 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG { 10:43:32 DEBUG { 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "instance": "10.142.0.24:9100" 10:43:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 DEBUG [...] 10:43:32 DEBUG }, 10:43:32 DEBUG }, 10:43:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG "value": [ 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG "value": [ 10:43:32 DEBUG }, 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG 10:43:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:43:32 DEBUG 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG 10:43:32 DEBUG "0" 10:43:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:43:32 DEBUG }, 10:43:32 DEBUG [...] 10:43:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:43:32 DEBUG [...] 10:43:32 DEBUG ] 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG } 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG ] 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "0" 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG } 10:43:32 DEBUG "0" 10:43:32 DEBUG ] 10:43:32 DEBUG "instance": "10.142.0.74:9100" 10:43:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG }, 10:43:32 DEBUG }, 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG "value": [ 10:43:32 DEBUG "value": [ 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 DEBUG "0.00030000000000001136" 10:43:32 DEBUG "0" 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "0" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "0" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "pod": "ds-idrepo-1" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "pod": "ds-idrepo-0" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "698.3" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG { 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:43:32 DEBUG "data": { 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG { 10:43:32 DEBUG "result": [ 10:43:32 DEBUG "status": "success", 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG { 10:43:32 DEBUG "data": { 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:43:32 DEBUG "result": [ 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG }, 10:43:32 DEBUG { 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:43:32 DEBUG "value": [ 10:43:32 DEBUG "metric": { 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:43:32 DEBUG { 10:43:32 DEBUG 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG }, 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG "value": [ 10:43:32 DEBUG "data": { 10:43:32 DEBUG [...] 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:43:32 DEBUG 10:43:32 DEBUG "result": [ 10:43:32 DEBUG 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG 10:43:32 DEBUG { 10:43:32 DEBUG 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG [...] 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG 10:43:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:43:32 DEBUG }, 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:43:32 DEBUG 10:43:32 DEBUG "0.000012689691261118433" 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG }, 10:43:32 DEBUG 10:43:32 DEBUG ] 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG } 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "instance": "10.142.0.24:9100" 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG [...] 10:43:32 DEBUG ] 10:43:32 DEBUG "pod": "ds-cts-1" 10:43:32 DEBUG }, 10:43:32 DEBUG "0.2910370059770311" 10:43:32 DEBUG 10:43:32 DEBUG } 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG ] 10:43:32 DEBUG 10:43:32 DEBUG } 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG } 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG ] 10:43:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG } 10:43:32 DEBUG }, 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG } 10:43:32 DEBUG "value": [ 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "0" 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG ] 10:43:32 DEBUG 10:43:32 DEBUG "instance": "10.142.0.74:9100" 10:43:32 DEBUG } 10:43:32 DEBUG "pod": "ds-cts-0" 10:43:32 DEBUG }, 10:43:32 DEBUG ] 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG } 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG } 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "6508.633333333333" 10:43:32 DEBUG "0" 10:43:32 DEBUG ] 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951812" 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "0.015" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "pod": "ds-idrepo-1" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "pod": "ds-idrepo-0" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "698.2666666666667" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "1324048384" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "instance": "10.142.0.24:9100" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "instance": "10.142.0.74:9100" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "2889264128" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG } 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "pod": "ds-idrepo-1" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "pod": "ds-idrepo-0" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "698.2" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "0" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951812" 10:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951812" 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "instance": "10.142.0.24:9100" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "instance": "10.142.0.74:9100" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "0.00033333333333333365" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:32 INFO [http_cmd]: http status code OK 10:43:32 DEBUG --- status code --- 10:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:43:32 DEBUG --- http response --- 10:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:43:32 DEBUG ----- output ----- 10:43:32 DEBUG { 10:43:32 DEBUG "status": "success", 10:43:32 DEBUG "data": { 10:43:32 DEBUG "resultType": "vector", 10:43:32 DEBUG "result": [ 10:43:32 DEBUG { 10:43:32 DEBUG "metric": { 10:43:32 DEBUG "pod": "ds-idrepo-1" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG [...] 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG 10:43:32 DEBUG "pod": "ds-cts-0" 10:43:32 DEBUG }, 10:43:32 DEBUG "value": [ 10:43:32 DEBUG 1713951812, 10:43:32 DEBUG "0" 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG ] 10:43:32 DEBUG } 10:43:32 DEBUG } 10:43:43 INFO 10:43:43 INFO [loop_until]: kubectl --namespace=xlou top pods 10:43:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:43:43 INFO [loop_until]: OK (rc = 0) 10:43:43 DEBUG --- stdout --- 10:43:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 3m 3466Mi am-769b98f8d5-2rr6f 4m 3409Mi am-769b98f8d5-h2r87 6m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 6m 542Mi ds-cts-2 7m 503Mi ds-idrepo-0 2845m 11800Mi ds-idrepo-1 2719m 11822Mi ds-idrepo-2 9m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 6120m 4499Mi idm-6c95d7588f-j6s6c 3027m 3182Mi lodemon-6b9df475cd-csxlf 1m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 303m 1168Mi 10:43:43 DEBUG --- stderr --- 10:43:43 DEBUG 10:43:48 INFO 10:43:48 INFO [loop_until]: kubectl --namespace=xlou top node 10:43:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:43:48 INFO [loop_until]: OK (rc = 0) 10:43:48 DEBUG --- stdout --- 10:43:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 71m 0% 8235Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 2060m 12% 13091Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 2833m 17% 13096Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 60m 0% 1757Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 353m 2% 6142Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 61m 0% 1722Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 57m 0% 1638Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 82m 0% 4980Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 5403m 34% 6068Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 143m 0% 2462Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 64m 0% 4827Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 58m 0% 1181Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3451m 21% 4734Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 344m 2% 4598Mi 7% 10:43:48 DEBUG --- stderr --- 10:43:48 DEBUG 10:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951871" 10:44:31 INFO [http_cmd]: http status code OK 10:44:31 DEBUG --- status code --- 10:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:31 DEBUG --- http response --- 10:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:44:31 DEBUG ----- output ----- 10:44:31 DEBUG { 10:44:31 DEBUG "status": "success", 10:44:31 DEBUG "data": { 10:44:31 DEBUG "resultType": "vector", 10:44:31 DEBUG "result": [ 10:44:31 DEBUG { 10:44:31 DEBUG "metric": { 10:44:31 DEBUG "pod": "ds-cts-1" 10:44:31 DEBUG }, 10:44:31 DEBUG "value": [ 10:44:31 DEBUG 10:44:31 DEBUG 10:44:31 DEBUG 10:44:31 DEBUG [...] 10:44:31 DEBUG 10:44:31 DEBUG 10:44:31 DEBUG 10:44:31 DEBUG "pod": "ds-cts-0" 10:44:31 DEBUG }, 10:44:31 DEBUG "value": [ 10:44:31 DEBUG 1713951871, 10:44:31 DEBUG "0" 10:44:31 DEBUG ] 10:44:31 DEBUG } 10:44:31 DEBUG ] 10:44:31 DEBUG } 10:44:31 DEBUG } 10:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951871" 10:44:31 ERROR [http_cmd]: ERROR 10:44:31 ERROR --- status code --- 10:44:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:44:31 ERROR --- http response --- 10:44:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:44:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951871" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "0" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "instance": "10.142.0.24:9100" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "instance": "10.142.0.74:9100" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951871, 10:44:32 DEBUG "12051.666666666666" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG { 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "data": { 10:44:32 DEBUG "pod": "ds-cts-1" 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG }, 10:44:32 DEBUG "result": [ 10:44:32 DEBUG "value": [ 10:44:32 DEBUG { 10:44:32 DEBUG 10:44:32 DEBUG "metric": { 10:44:32 DEBUG 10:44:32 DEBUG "pod": "ds-idrepo-1" 10:44:32 DEBUG 10:44:32 DEBUG }, 10:44:32 DEBUG [...] 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "ds-cts-0" 10:44:32 DEBUG [...] 10:44:32 DEBUG }, 10:44:32 DEBUG 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG 10:44:32 DEBUG "0" 10:44:32 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:44:32 DEBUG ] 10:44:32 DEBUG }, 10:44:32 DEBUG } 10:44:32 DEBUG "value": [ 10:44:32 DEBUG ] 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG } 10:44:32 DEBUG "9528.621249949174" 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "0" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "pod": "ds-idrepo-1" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "0" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:44:32 DEBUG { 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:44:32 DEBUG { 10:44:32 DEBUG "data": { 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:44:32 DEBUG { 10:44:32 DEBUG { 10:44:32 DEBUG "data": { 10:44:32 DEBUG "result": [ 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG { 10:44:32 DEBUG { 10:44:32 DEBUG "data": { 10:44:32 DEBUG "data": { 10:44:32 DEBUG "result": [ 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG { 10:44:32 DEBUG "pod": "ds-idrepo-1" 10:44:32 DEBUG "data": { 10:44:32 DEBUG "result": [ 10:44:32 DEBUG "result": [ 10:44:32 DEBUG "metric": { 10:44:32 DEBUG }, 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG { 10:44:32 DEBUG { 10:44:32 DEBUG "instance": "10.142.0.24:9100" 10:44:32 DEBUG "value": [ 10:44:32 DEBUG "result": [ 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG }, 10:44:32 DEBUG 10:44:32 DEBUG { 10:44:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:44:32 DEBUG "pod": "ds-idrepo-1" 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG "metric": { 10:44:32 DEBUG }, 10:44:32 DEBUG }, 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:44:32 DEBUG "value": [ 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG }, 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG "pod": "ds-idrepo-0" 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:44:32 DEBUG }, 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "instance": "10.142.0.74:9100" 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG }, 10:44:32 DEBUG { 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG 10:44:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:44:32 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:44:32 DEBUG "value": [ 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "665.5666666666666" 10:44:32 DEBUG 10:44:32 DEBUG }, 10:44:32 DEBUG }, 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "data": { 10:44:32 DEBUG ] 10:44:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:44:32 DEBUG "value": [ 10:44:32 DEBUG "value": [ 10:44:32 DEBUG "0.00030000000000001136" 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG } 10:44:32 DEBUG }, 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG ] 10:44:32 DEBUG "result": [ 10:44:32 DEBUG ] 10:44:32 DEBUG "value": [ 10:44:32 DEBUG "0.28736918284240226" 10:44:32 DEBUG "0" 10:44:32 DEBUG } 10:44:32 DEBUG { 10:44:32 DEBUG } 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG ] 10:44:32 DEBUG ] 10:44:32 DEBUG ] 10:44:32 DEBUG "metric": { 10:44:32 DEBUG } 10:44:32 DEBUG "0" 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:44:32 DEBUG ] 10:44:32 DEBUG ] 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG }, 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 DEBUG "value": [ 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 DEBUG 10:44:32 DEBUG } 10:44:32 DEBUG 10:44:32 DEBUG } 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "0" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "pod": "ds-idrepo-1" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "0.160066826302851" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG "metric": { 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG }, 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:44:32 DEBUG [...] 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG 10:44:32 DEBUG { 10:44:32 DEBUG 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG 10:44:32 DEBUG "data": { 10:44:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG }, 10:44:32 DEBUG "result": [ 10:44:32 DEBUG "value": [ 10:44:32 DEBUG { 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "metric": { 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG "0" 10:44:32 DEBUG "pod": "ds-idrepo-1" 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG ] 10:44:32 DEBUG }, 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG } 10:44:32 DEBUG "value": [ 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:44:32 DEBUG ] 10:44:32 DEBUG 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG } 10:44:32 DEBUG 10:44:32 DEBUG { 10:44:32 DEBUG } 10:44:32 DEBUG 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG [...] 10:44:32 DEBUG "data": { 10:44:32 DEBUG 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG "result": [ 10:44:32 DEBUG 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG { 10:44:32 DEBUG "pod": "ds-idrepo-0" 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG "metric": { 10:44:32 DEBUG }, 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:44:32 DEBUG "pod": "ds-cts-1" 10:44:32 DEBUG "value": [ 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG }, 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG { 10:44:32 DEBUG "value": [ 10:44:32 DEBUG "665.5666666666666" 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG 10:44:32 DEBUG ] 10:44:32 DEBUG "data": { 10:44:32 DEBUG 10:44:32 DEBUG } 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG 10:44:32 DEBUG ] 10:44:32 DEBUG "result": [ 10:44:32 DEBUG [...] 10:44:32 DEBUG } 10:44:32 DEBUG { 10:44:32 DEBUG 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 DEBUG } 10:44:32 DEBUG "metric": { 10:44:32 DEBUG 10:44:32 DEBUG "instance": "10.142.0.24:9100" 10:44:32 DEBUG 10:44:32 DEBUG }, 10:44:32 DEBUG "pod": "ds-cts-0" 10:44:32 DEBUG "value": [ 10:44:32 DEBUG }, 10:44:32 DEBUG 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG 10:44:32 DEBUG "0" 10:44:32 DEBUG [...] 10:44:32 DEBUG ] 10:44:32 DEBUG 10:44:32 DEBUG } 10:44:32 DEBUG 10:44:32 DEBUG ] 10:44:32 DEBUG 10:44:32 DEBUG } 10:44:32 DEBUG "instance": "10.142.0.74:9100" 10:44:32 DEBUG } 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "6844.700000000001" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951872" 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "0.015" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "pod": "ds-idrepo-1" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "ds-idrepo-0" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "665.5666666666666" 10:44:32 DEBUG ] 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG } 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG ] 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG } 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG } 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "instance": "10.142.0.24:9100" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "instance": "10.142.0.74:9100" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "2889579520" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "1325182976" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951872" 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "pod": "ds-idrepo-1" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "ds-cts-0" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "0" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "0" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951872" 10:44:32 INFO [http_cmd]: http status code OK 10:44:32 DEBUG --- status code --- 10:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:44:32 DEBUG --- http response --- 10:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:44:32 DEBUG ----- output ----- 10:44:32 DEBUG { 10:44:32 DEBUG "status": "success", 10:44:32 DEBUG "data": { 10:44:32 DEBUG "resultType": "vector", 10:44:32 DEBUG "result": [ 10:44:32 DEBUG { 10:44:32 DEBUG "metric": { 10:44:32 DEBUG "instance": "10.142.0.24:9100" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG [...] 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG 10:44:32 DEBUG "instance": "10.142.0.74:9100" 10:44:32 DEBUG }, 10:44:32 DEBUG "value": [ 10:44:32 DEBUG 1713951872, 10:44:32 DEBUG "0" 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG ] 10:44:32 DEBUG } 10:44:32 DEBUG } 10:44:43 INFO 10:44:43 INFO [loop_until]: kubectl --namespace=xlou top pods 10:44:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:44:43 INFO [loop_until]: OK (rc = 0) 10:44:43 DEBUG --- stdout --- 10:44:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 7m 3466Mi am-769b98f8d5-2rr6f 6m 3409Mi am-769b98f8d5-h2r87 5m 3435Mi ds-cts-0 5m 563Mi ds-cts-1 6m 542Mi ds-cts-2 5m 503Mi ds-idrepo-0 2903m 11807Mi ds-idrepo-1 2187m 11825Mi ds-idrepo-2 15m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 3702m 4501Mi idm-6c95d7588f-j6s6c 3928m 3186Mi lodemon-6b9df475cd-csxlf 5m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 293m 1168Mi 10:44:43 DEBUG --- stderr --- 10:44:43 DEBUG 10:44:48 INFO 10:44:48 INFO [loop_until]: kubectl --namespace=xlou top node 10:44:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:44:48 INFO [loop_until]: OK (rc = 0) 10:44:48 DEBUG --- stdout --- 10:44:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 75m 0% 8248Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 1678m 10% 13096Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 2665m 16% 13108Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 58m 0% 1759Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 341m 2% 6141Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1736Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 57m 0% 1639Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 85m 0% 4979Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4163m 26% 6070Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 148m 0% 2456Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 67m 0% 4831Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 56m 0% 1192Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3614m 22% 4740Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 333m 2% 4600Mi 7% 10:44:48 DEBUG --- stderr --- 10:44:48 DEBUG 10:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951931" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "ds-cts-1" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "pod": "ds-cts-0" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951931, 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951932" 10:45:32 ERROR [http_cmd]: ERROR 10:45:32 ERROR --- status code --- 10:45:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:45:32 ERROR --- http response --- 10:45:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:45:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "instance": "10.142.0.24:9100" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "instance": "10.142.0.74:9100" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "13408.433333333332" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:45:32 DEBUG "pod": "ds-idrepo-1" 10:45:32 DEBUG }, 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:45:32 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:45:32 DEBUG }, 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0" 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "ds-cts-1" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "pod": "ds-cts-0" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "ds-idrepo-1" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951932" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "ds-idrepo-1" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "pod": "ds-idrepo-0" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "850.9333333333333" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "ds-idrepo-1" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0.2862608710699744" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951932" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG { 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG "data": { 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG { 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG }, 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "ds-idrepo-1" 10:45:32 DEBUG 10:45:32 DEBUG "instance": "10.142.0.24:9100" 10:45:32 DEBUG }, 10:45:32 DEBUG 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG }, 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "instance": "10.142.0.74:9100" 10:45:32 DEBUG }, 10:45:32 DEBUG "0" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG ] 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG } 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG "0.00013333333333337785" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951932" 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "ds-idrepo-1" 10:45:32 DEBUG }, 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG "value": [ 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:45:32 DEBUG [...] 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG { 10:45:32 DEBUG 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "data": { 10:45:32 DEBUG "pod": "ds-idrepo-0" 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG }, 10:45:32 DEBUG "resultType": "vector", 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951932" 10:45:32 DEBUG "result": [ 10:45:32 DEBUG "result": [ 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG { 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "850.9666666666667" 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "ds-idrepo-1" 10:45:32 DEBUG "pod": "ds-idrepo-1" 10:45:32 DEBUG ] 10:45:32 DEBUG "instance": "10.142.0.24:9100" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG }, 10:45:32 DEBUG }, 10:45:32 DEBUG } 10:45:32 DEBUG }, 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG "value": [ 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG "value": [ 10:45:32 DEBUG ] 10:45:32 DEBUG "value": [ 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG 10:45:32 DEBUG } 10:45:32 DEBUG 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG 10:45:32 DEBUG } 10:45:32 DEBUG 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:45:32 DEBUG 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG [...] 10:45:32 DEBUG [...] 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:45:32 DEBUG [...] 10:45:32 DEBUG { 10:45:32 DEBUG { 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG { 10:45:32 DEBUG 10:45:32 DEBUG "data": { 10:45:32 DEBUG "data": { 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "pod": "ds-idrepo-0" 10:45:32 DEBUG "pod": "ds-cts-0" 10:45:32 DEBUG "data": { 10:45:32 DEBUG "instance": "10.142.0.74:9100" 10:45:32 DEBUG "result": [ 10:45:32 DEBUG "result": [ 10:45:32 DEBUG }, 10:45:32 DEBUG }, 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG }, 10:45:32 DEBUG { 10:45:32 DEBUG { 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "result": [ 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG { 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "pod": "ds-cts-1" 10:45:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:45:32 DEBUG "851" 10:45:32 DEBUG "0" 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "6718.733333333334" 10:45:32 DEBUG }, 10:45:32 DEBUG }, 10:45:32 DEBUG ] 10:45:32 DEBUG ] 10:45:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:45:32 DEBUG ] 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "value": [ 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG }, 10:45:32 DEBUG } 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG ] 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG ] 10:45:32 DEBUG "value": [ 10:45:32 DEBUG ] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG } 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG } 10:45:32 DEBUG 10:45:32 DEBUG } 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG } 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG } 10:45:32 DEBUG 10:45:32 DEBUG } 10:45:32 DEBUG [...] 10:45:32 DEBUG [...] 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG { 10:45:32 DEBUG 10:45:32 DEBUG "pod": "ds-cts-0" 10:45:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG 10:45:32 DEBUG }, 10:45:32 DEBUG }, 10:45:32 DEBUG "data": { 10:45:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG }, 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "result": [ 10:45:32 DEBUG "value": [ 10:45:32 DEBUG "0" 10:45:32 DEBUG "0.015" 10:45:32 DEBUG { 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG ] 10:45:32 DEBUG ] 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "1324511232" 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG "instance": "10.142.0.24:9100" 10:45:32 DEBUG ] 10:45:32 DEBUG ] 10:45:32 DEBUG ] 10:45:32 DEBUG }, 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG "value": [ 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 DEBUG 10:45:32 DEBUG } 10:45:32 DEBUG 10:45:32 DEBUG } 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "instance": "10.142.0.74:9100" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "2889858048" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951932" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951932" 10:45:32 INFO [http_cmd]: http status code OK 10:45:32 DEBUG --- status code --- 10:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:45:32 DEBUG --- http response --- 10:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:45:32 DEBUG ----- output ----- 10:45:32 DEBUG { 10:45:32 DEBUG "status": "success", 10:45:32 DEBUG "data": { 10:45:32 DEBUG "resultType": "vector", 10:45:32 DEBUG "result": [ 10:45:32 DEBUG { 10:45:32 DEBUG "metric": { 10:45:32 DEBUG "instance": "10.142.0.24:9100" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG [...] 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG 10:45:32 DEBUG "instance": "10.142.0.74:9100" 10:45:32 DEBUG }, 10:45:32 DEBUG "value": [ 10:45:32 DEBUG 1713951932, 10:45:32 DEBUG "0" 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG ] 10:45:32 DEBUG } 10:45:32 DEBUG } 10:45:43 INFO 10:45:43 INFO [loop_until]: kubectl --namespace=xlou top pods 10:45:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:45:43 INFO [loop_until]: OK (rc = 0) 10:45:43 DEBUG --- stdout --- 10:45:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 4m 3408Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 563Mi ds-cts-1 6m 543Mi ds-cts-2 11m 503Mi ds-idrepo-0 4359m 11790Mi ds-idrepo-1 11m 11825Mi ds-idrepo-2 7m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 3889m 4500Mi idm-6c95d7588f-j6s6c 3779m 3191Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 283m 1168Mi 10:45:43 DEBUG --- stderr --- 10:45:43 DEBUG 10:45:48 INFO 10:45:48 INFO [loop_until]: kubectl --namespace=xlou top node 10:45:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:45:48 INFO [loop_until]: OK (rc = 0) 10:45:48 DEBUG --- stdout --- 10:45:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 67m 0% 8247Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 404m 2% 13090Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4257m 26% 13104Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 58m 0% 1765Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 351m 2% 6146Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1723Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 61m 0% 1637Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 81m 0% 4972Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4605m 28% 6068Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 130m 0% 2460Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 68m 0% 4829Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 58m 0% 1197Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 4283m 26% 4840Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 339m 2% 4599Mi 7% 10:45:48 DEBUG --- stderr --- 10:45:48 DEBUG 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "pod": "ds-cts-1" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "ds-cts-0" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 ERROR [http_cmd]: ERROR 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 ERROR --- status code --- 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 ERROR --- http response --- 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:46:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "pod": "ds-cts-1" 10:46:32 DEBUG "instance": "10.142.0.24:9100" 10:46:32 DEBUG }, 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "ds-cts-0" 10:46:32 DEBUG "instance": "10.142.0.74:9100" 10:46:32 DEBUG }, 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0" 10:46:32 DEBUG "12153.066666666666" 10:46:32 DEBUG ] 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "pod": "ds-idrepo-1" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG --- status code --- 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:46:32 DEBUG { 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG { 10:46:32 DEBUG { 10:46:32 DEBUG "data": { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "data": { 10:46:32 DEBUG "result": [ 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG { 10:46:32 DEBUG "result": [ 10:46:32 DEBUG "result": [ 10:46:32 DEBUG "metric": { 10:46:32 DEBUG { 10:46:32 DEBUG { 10:46:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG }, 10:46:32 DEBUG "pod": "ds-idrepo-1" 10:46:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:46:32 DEBUG "value": [ 10:46:32 DEBUG }, 10:46:32 DEBUG }, 10:46:32 DEBUG 10:46:32 DEBUG "value": [ 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG }, 10:46:32 DEBUG "pod": "ds-idrepo-0" 10:46:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:46:32 DEBUG "value": [ 10:46:32 DEBUG }, 10:46:32 DEBUG }, 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG "value": [ 10:46:32 DEBUG "0" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG ] 10:46:32 DEBUG "554.9666666666667" 10:46:32 DEBUG "0" 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG ] 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "pod": "ds-idrepo-1" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "pod": "ds-idrepo-1" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713951992" 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "pod": "ds-idrepo-1" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0.000023327876520111245" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0.31269801225142535" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG { 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:46:32 DEBUG { 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG "metric": { 10:46:32 DEBUG { 10:46:32 DEBUG "pod": "ds-idrepo-1" 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG }, 10:46:32 DEBUG "data": { 10:46:32 DEBUG "value": [ 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG 10:46:32 DEBUG "result": [ 10:46:32 DEBUG 10:46:32 DEBUG { 10:46:32 DEBUG 10:46:32 DEBUG "metric": { 10:46:32 DEBUG [...] 10:46:32 DEBUG "instance": "10.142.0.24:9100" 10:46:32 DEBUG 10:46:32 DEBUG }, 10:46:32 DEBUG 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "ds-idrepo-0" 10:46:32 DEBUG 10:46:32 DEBUG }, 10:46:32 DEBUG 10:46:32 DEBUG "value": [ 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG [...] 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG 10:46:32 DEBUG "554.9666666666667" 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG 10:46:32 DEBUG ] 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG 10:46:32 DEBUG } 10:46:32 DEBUG "instance": "10.142.0.74:9100" 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:46:32 DEBUG ] 10:46:32 DEBUG }, 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG } 10:46:32 DEBUG "value": [ 10:46:32 DEBUG { 10:46:32 DEBUG } 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "0.0002666666666667557" 10:46:32 DEBUG "data": { 10:46:32 DEBUG ] 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG } 10:46:32 DEBUG "result": [ 10:46:32 DEBUG ] 10:46:32 DEBUG { 10:46:32 DEBUG } 10:46:32 DEBUG "metric": { 10:46:32 DEBUG } 10:46:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713951992" 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG ----- output ----- 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:46:32 DEBUG { 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG { 10:46:32 DEBUG "data": { 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:46:32 DEBUG "data": { 10:46:32 DEBUG "result": [ 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG { 10:46:32 DEBUG { 10:46:32 DEBUG "result": [ 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG { 10:46:32 DEBUG "pod": "ds-idrepo-1" 10:46:32 DEBUG "data": { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG }, 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:46:32 DEBUG "value": [ 10:46:32 DEBUG "result": [ 10:46:32 DEBUG }, 10:46:32 DEBUG 10:46:32 DEBUG { 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG "metric": { 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "ds-cts-1" 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG }, 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "value": [ 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "ds-cts-0" 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG }, 10:46:32 DEBUG [...] 10:46:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG }, 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG 10:46:32 DEBUG "value": [ 10:46:32 DEBUG "0" 10:46:32 DEBUG 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG ] 10:46:32 DEBUG "pod": "ds-cts-0" 10:46:32 DEBUG "0.015" 10:46:32 DEBUG } 10:46:32 DEBUG }, 10:46:32 DEBUG ] 10:46:32 DEBUG ] 10:46:32 DEBUG "value": [ 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951992" 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG "0" 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG } 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG } 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951992" 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713951992" 10:46:32 DEBUG "pod": "ds-idrepo-1" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "pod": "ds-idrepo-0" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "554.9666666666667" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG "data": { 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG "result": [ 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:46:32 DEBUG { 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG "metric": { 10:46:32 DEBUG { 10:46:32 DEBUG "instance": "10.142.0.24:9100" 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG }, 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG "data": { 10:46:32 DEBUG "value": [ 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG "result": [ 10:46:32 DEBUG 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG { 10:46:32 DEBUG 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:46:32 DEBUG "metric": { 10:46:32 DEBUG [...] 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG "instance": "10.142.0.24:9100" 10:46:32 DEBUG 10:46:32 DEBUG { 10:46:32 DEBUG }, 10:46:32 DEBUG 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG "data": { 10:46:32 DEBUG 10:46:32 DEBUG "instance": "10.142.0.74:9100" 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG 10:46:32 DEBUG }, 10:46:32 DEBUG "result": [ 10:46:32 DEBUG 10:46:32 DEBUG "value": [ 10:46:32 DEBUG { 10:46:32 DEBUG [...] 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "metric": { 10:46:32 DEBUG 10:46:32 DEBUG "8244.466666666667" 10:46:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:46:32 DEBUG 10:46:32 DEBUG ] 10:46:32 DEBUG }, 10:46:32 DEBUG 10:46:32 DEBUG } 10:46:32 DEBUG "value": [ 10:46:32 DEBUG "instance": "10.142.0.74:9100" 10:46:32 DEBUG ] 10:46:32 DEBUG 10:46:32 DEBUG }, 10:46:32 DEBUG } 10:46:32 DEBUG 10:46:32 DEBUG "value": [ 10:46:32 DEBUG } 10:46:32 DEBUG 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG [...] 10:46:32 DEBUG "2890081280" 10:46:32 DEBUG 10:46:32 DEBUG ] 10:46:32 DEBUG 10:46:32 DEBUG } 10:46:32 DEBUG 10:46:32 DEBUG ] 10:46:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:46:32 DEBUG } 10:46:32 DEBUG }, 10:46:32 DEBUG } 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "1325363200" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713951992" 10:46:32 INFO [http_cmd]: http status code OK 10:46:32 DEBUG --- status code --- 10:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:46:32 DEBUG --- http response --- 10:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:46:32 DEBUG ----- output ----- 10:46:32 DEBUG { 10:46:32 DEBUG "status": "success", 10:46:32 DEBUG "data": { 10:46:32 DEBUG "resultType": "vector", 10:46:32 DEBUG "result": [ 10:46:32 DEBUG { 10:46:32 DEBUG "metric": { 10:46:32 DEBUG "instance": "10.142.0.24:9100" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG [...] 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG 10:46:32 DEBUG "instance": "10.142.0.74:9100" 10:46:32 DEBUG }, 10:46:32 DEBUG "value": [ 10:46:32 DEBUG 1713951992, 10:46:32 DEBUG "0.00033333333333333365" 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG ] 10:46:32 DEBUG } 10:46:32 DEBUG } 10:46:43 INFO 10:46:43 INFO [loop_until]: kubectl --namespace=xlou top pods 10:46:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:46:43 INFO [loop_until]: OK (rc = 0) 10:46:43 DEBUG --- stdout --- 10:46:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 4m 3408Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 6m 543Mi ds-cts-2 5m 503Mi ds-idrepo-0 4005m 11795Mi ds-idrepo-1 12m 11829Mi ds-idrepo-2 9m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 3716m 4501Mi idm-6c95d7588f-j6s6c 3214m 3303Mi lodemon-6b9df475cd-csxlf 4m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 265m 1168Mi 10:46:43 DEBUG --- stderr --- 10:46:43 DEBUG 10:46:48 INFO 10:46:48 INFO [loop_until]: kubectl --namespace=xlou top node 10:46:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:46:48 INFO [loop_until]: OK (rc = 0) 10:46:48 DEBUG --- stdout --- 10:46:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 68m 0% 8240Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 56m 0% 13103Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4184m 26% 13108Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 58m 0% 1761Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 339m 2% 6136Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 64m 0% 1718Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1636Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 84m 0% 4971Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 3640m 22% 6058Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 147m 0% 2456Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 65m 0% 4820Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 62m 0% 1190Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3268m 20% 4852Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 328m 2% 4601Mi 7% 10:46:48 DEBUG --- stderr --- 10:46:48 DEBUG 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "ds-cts-1" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "ds-cts-0" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 ERROR [http_cmd]: ERROR 10:47:32 ERROR --- status code --- 10:47:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:47:32 ERROR --- http response --- 10:47:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:47:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:47:32 DEBUG "instance": "10.142.0.24:9100" 10:47:32 DEBUG }, 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "ds-cts-2" 10:47:32 DEBUG "instance": "10.142.0.74:9100" 10:47:32 DEBUG }, 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "18076826.780529466" 10:47:32 DEBUG "12919.533333333333" 10:47:32 DEBUG ] 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "ds-cts-1" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "ds-cts-0" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:47:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:47:32 DEBUG }, 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:47:32 DEBUG }, 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG "0" 10:47:32 DEBUG ] 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:47:32 DEBUG { 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG { 10:47:32 DEBUG "data": { 10:47:32 DEBUG { 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 DEBUG "data": { 10:47:32 DEBUG "result": [ 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "pod": "ds-idrepo-1" 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG }, 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:47:32 DEBUG "value": [ 10:47:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:47:32 DEBUG }, 10:47:32 DEBUG 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "ds-idrepo-0" 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG }, 10:47:32 DEBUG 10:47:32 DEBUG "pod": "ds-cts-2" 10:47:32 DEBUG "value": [ 10:47:32 DEBUG "pod": "ds-cts-2" 10:47:32 DEBUG }, 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG "639.3666666666667" 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG ] 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG } 10:47:32 DEBUG "699.5730145175065" 10:47:32 DEBUG ] 10:47:32 DEBUG ] 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952052" 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "ds-cts-2" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0.0043603074295471755" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG { 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG "data": { 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "data": { 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG "instance": "10.142.0.24:9100" 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG }, 10:47:32 DEBUG "result": [ 10:47:32 DEBUG "value": [ 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:47:32 DEBUG { 10:47:32 DEBUG 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG "metric": { 10:47:32 DEBUG 10:47:32 DEBUG { 10:47:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:47:32 DEBUG 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG }, 10:47:32 DEBUG [...] 10:47:32 DEBUG "data": { 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "result": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG { 10:47:32 DEBUG 10:47:32 DEBUG "instance": "10.142.0.74:9100" 10:47:32 DEBUG "metric": { 10:47:32 DEBUG [...] 10:47:32 DEBUG }, 10:47:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:47:32 DEBUG 10:47:32 DEBUG "value": [ 10:47:32 DEBUG }, 10:47:32 DEBUG 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG "0.00013333333333325942" 10:47:32 DEBUG 10:47:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:47:32 DEBUG ] 10:47:32 DEBUG 10:47:32 DEBUG }, 10:47:32 DEBUG } 10:47:32 DEBUG 10:47:32 DEBUG "value": [ 10:47:32 DEBUG ] 10:47:32 DEBUG [...] 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG } 10:47:32 DEBUG 10:47:32 DEBUG "0.29757352153161665" 10:47:32 DEBUG } 10:47:32 DEBUG 10:47:32 DEBUG ] 10:47:32 DEBUG 10:47:32 DEBUG } 10:47:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:47:32 DEBUG ] 10:47:32 DEBUG }, 10:47:32 DEBUG } 10:47:32 DEBUG "value": [ 10:47:32 DEBUG } 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0.015" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "ds-idrepo-1" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "ds-idrepo-0" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "639.3333333333334" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952052" 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG { 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG "metric": { 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG "pod": "ds-idrepo-1" 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG }, 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:47:32 DEBUG "value": [ 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG 10:47:32 DEBUG { 10:47:32 DEBUG 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG 10:47:32 DEBUG "data": { 10:47:32 DEBUG [...] 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG 10:47:32 DEBUG "result": [ 10:47:32 DEBUG 10:47:32 DEBUG { 10:47:32 DEBUG 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "ds-idrepo-0" 10:47:32 DEBUG "instance": "10.142.0.24:9100" 10:47:32 DEBUG }, 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG 10:47:32 DEBUG "639.3666666666667" 10:47:32 DEBUG 10:47:32 DEBUG ] 10:47:32 DEBUG 10:47:32 DEBUG } 10:47:32 DEBUG [...] 10:47:32 DEBUG ] 10:47:32 DEBUG 10:47:32 DEBUG } 10:47:32 DEBUG 10:47:32 DEBUG } 10:47:32 DEBUG 10:47:32 DEBUG "instance": "10.142.0.74:9100" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "7871" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952052" 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "ds-cts-1" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "ds-cts-0" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "1323737088" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952052" 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "pod": "ds-idrepo-1" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "pod": "ds-cts-0" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "instance": "10.142.0.24:9100" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "instance": "10.142.0.74:9100" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "2890396672" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952052" 10:47:32 INFO [http_cmd]: http status code OK 10:47:32 DEBUG --- status code --- 10:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:47:32 DEBUG --- http response --- 10:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:47:32 DEBUG ----- output ----- 10:47:32 DEBUG { 10:47:32 DEBUG "status": "success", 10:47:32 DEBUG "data": { 10:47:32 DEBUG "resultType": "vector", 10:47:32 DEBUG "result": [ 10:47:32 DEBUG { 10:47:32 DEBUG "metric": { 10:47:32 DEBUG "instance": "10.142.0.24:9100" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG [...] 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG 10:47:32 DEBUG "instance": "10.142.0.74:9100" 10:47:32 DEBUG }, 10:47:32 DEBUG "value": [ 10:47:32 DEBUG 1713952052, 10:47:32 DEBUG "0" 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG ] 10:47:32 DEBUG } 10:47:32 DEBUG } 10:47:43 INFO 10:47:43 INFO [loop_until]: kubectl --namespace=xlou top pods 10:47:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:47:43 INFO [loop_until]: OK (rc = 0) 10:47:43 DEBUG --- stdout --- 10:47:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 3m 3466Mi am-769b98f8d5-2rr6f 5m 3408Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 562Mi ds-cts-1 6m 542Mi ds-cts-2 5m 503Mi ds-idrepo-0 3031m 11791Mi ds-idrepo-1 1989m 11833Mi ds-idrepo-2 8m 5819Mi end-user-ui-58878d7668-zkqbp 1m 4Mi idm-6c95d7588f-98n7b 6142m 4503Mi idm-6c95d7588f-j6s6c 3929m 3307Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 301m 1168Mi 10:47:43 DEBUG --- stderr --- 10:47:43 DEBUG 10:47:48 INFO 10:47:48 INFO [loop_until]: kubectl --namespace=xlou top node 10:47:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:47:48 INFO [loop_until]: OK (rc = 0) 10:47:48 DEBUG --- stdout --- 10:47:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 72m 0% 8249Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 2193m 13% 13111Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 2935m 18% 13096Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 62m 0% 1765Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 344m 2% 6133Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1720Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 58m 0% 1651Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 84m 0% 4972Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 5693m 35% 6073Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 151m 0% 2474Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 65m 0% 4823Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 60m 0% 1181Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3724m 23% 4861Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 374m 2% 4596Mi 7% 10:47:48 DEBUG --- stderr --- 10:47:48 DEBUG 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "ds-cts-1" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "ds-cts-0" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "instance": "10.142.0.24:9100" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "instance": "10.142.0.74:9100" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "11725.933333333332" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:48:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:48:32 DEBUG }, 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:48:32 DEBUG "pod": "idm-6c95d7588f-j6s6c" 10:48:32 DEBUG }, 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0" 10:48:32 DEBUG "147326010.0874774" 10:48:32 DEBUG ] 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "idm-6c95d7588f-j6s6c" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "ds-cts-1" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "ds-cts-0" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 ERROR [http_cmd]: ERROR 10:48:32 ERROR --- status code --- 10:48:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:48:32 ERROR --- http response --- 10:48:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:48:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG 10:48:32 DEBUG --- status code --- 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG }, 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG "value": [ 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:48:32 DEBUG "0" 10:48:32 DEBUG { 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG ] 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG { 10:48:32 DEBUG } 10:48:32 DEBUG "data": { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG ] 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "data": { 10:48:32 DEBUG } 10:48:32 DEBUG "result": [ 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG } 10:48:32 DEBUG { 10:48:32 DEBUG "result": [ 10:48:32 DEBUG "metric": { 10:48:32 DEBUG { 10:48:32 DEBUG "pod": "ds-idrepo-1" 10:48:32 DEBUG "metric": { 10:48:32 DEBUG }, 10:48:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:48:32 DEBUG "value": [ 10:48:32 DEBUG }, 10:48:32 DEBUG 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "ds-idrepo-0" 10:48:32 DEBUG 10:48:32 DEBUG }, 10:48:32 DEBUG "pod": "idm-6c95d7588f-j6s6c" 10:48:32 DEBUG "value": [ 10:48:32 DEBUG }, 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG "610.5666666666666" 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG ] 10:48:32 DEBUG "1074.9223857474826" 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 DEBUG "instance": "10.142.0.24:9100" 10:48:32 DEBUG }, 10:48:32 DEBUG }, 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG "value": [ 10:48:32 DEBUG "value": [ 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:48:32 DEBUG [...] 10:48:32 DEBUG [...] 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG { 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "data": { 10:48:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:48:32 DEBUG "instance": "10.142.0.74:9100" 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG }, 10:48:32 DEBUG }, 10:48:32 DEBUG "result": [ 10:48:32 DEBUG "value": [ 10:48:32 DEBUG "value": [ 10:48:32 DEBUG { 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "0" 10:48:32 DEBUG "0.00023333333333338166" 10:48:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:48:32 DEBUG ] 10:48:32 DEBUG ] 10:48:32 DEBUG }, 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG "value": [ 10:48:32 DEBUG ] 10:48:32 DEBUG ] 10:48:32 DEBUG 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "idm-6c95d7588f-j6s6c" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "6.968970524632163" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0.2976461653646823" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:48:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:48:32 DEBUG }, 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:48:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:48:32 DEBUG }, 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0.015" 10:48:32 DEBUG "0" 10:48:32 DEBUG ] 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "ds-idrepo-1" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "ds-idrepo-0" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "610.5666666666666" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "ds-cts-1" 10:48:32 DEBUG "pod": "ds-idrepo-1" 10:48:32 DEBUG }, 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "ds-cts-0" 10:48:32 DEBUG "pod": "ds-idrepo-0" 10:48:32 DEBUG }, 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0" 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG "610.5666666666666" 10:48:32 DEBUG ] 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG { 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG } 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "instance": "10.142.0.24:9100" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "instance": "10.142.0.74:9100" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "6781.433333333334" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952112" 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "1323315200" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "pod": "ds-idrepo-1" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "pod": "ds-cts-0" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "instance": "10.142.0.24:9100" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "instance": "10.142.0.74:9100" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "2890558464" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952112" 10:48:32 INFO [http_cmd]: http status code OK 10:48:32 DEBUG --- status code --- 10:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:48:32 DEBUG --- http response --- 10:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:48:32 DEBUG ----- output ----- 10:48:32 DEBUG { 10:48:32 DEBUG "status": "success", 10:48:32 DEBUG "data": { 10:48:32 DEBUG "resultType": "vector", 10:48:32 DEBUG "result": [ 10:48:32 DEBUG { 10:48:32 DEBUG "metric": { 10:48:32 DEBUG "instance": "10.142.0.24:9100" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG [...] 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG 10:48:32 DEBUG "instance": "10.142.0.74:9100" 10:48:32 DEBUG }, 10:48:32 DEBUG "value": [ 10:48:32 DEBUG 1713952112, 10:48:32 DEBUG "0.0006666666666666673" 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG ] 10:48:32 DEBUG } 10:48:32 DEBUG } 10:48:43 INFO 10:48:43 INFO [loop_until]: kubectl --namespace=xlou top pods 10:48:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:48:43 INFO [loop_until]: OK (rc = 0) 10:48:43 DEBUG --- stdout --- 10:48:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 4m 3409Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 15m 559Mi ds-cts-1 6m 542Mi ds-cts-2 5m 503Mi ds-idrepo-0 2941m 11795Mi ds-idrepo-1 2090m 11833Mi ds-idrepo-2 7m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4967m 4816Mi idm-6c95d7588f-j6s6c 3977m 3309Mi lodemon-6b9df475cd-csxlf 5m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 270m 1169Mi 10:48:43 DEBUG --- stderr --- 10:48:43 DEBUG 10:48:48 INFO 10:48:48 INFO [loop_until]: kubectl --namespace=xlou top node 10:48:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:48:49 INFO [loop_until]: OK (rc = 0) 10:48:49 DEBUG --- stdout --- 10:48:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 68m 0% 8239Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 1629m 10% 13113Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 2901m 18% 13103Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1759Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 332m 2% 6142Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 69m 0% 1717Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1626Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 82m 0% 4965Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4381m 27% 6372Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 146m 0% 2462Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 68m 0% 4826Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 61m 0% 1198Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3623m 22% 4868Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 352m 2% 4601Mi 7% 10:48:49 DEBUG --- stderr --- 10:48:49 DEBUG 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "ds-cts-1" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "ds-cts-0" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "instance": "10.142.0.24:9100" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "instance": "10.142.0.74:9100" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "11538.899999999998" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "ds-cts-2" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "17291456.944988895" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "ds-cts-2" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952172" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 ERROR [http_cmd]: ERROR 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:49:32 ERROR --- status code --- 10:49:32 DEBUG ----- output ----- 10:49:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:49:32 DEBUG { 10:49:32 ERROR --- http response --- 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:49:32 DEBUG "data": { 10:49:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "ds-cts-1" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "ds-cts-0" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "instance": "10.142.0.24:9100" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "instance": "10.142.0.74:9100" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0.00020000000000000757" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG ] 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG } 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG } 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG { 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG "data": { 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:49:32 DEBUG "resultType": "vector", 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG "result": [ 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG { 10:49:32 DEBUG { 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "metric": { 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG "data": { 10:49:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG }, 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG "result": [ 10:49:32 DEBUG "value": [ 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG { 10:49:32 DEBUG { 10:49:32 DEBUG 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "metric": { 10:49:32 DEBUG 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG "data": { 10:49:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:49:32 DEBUG 10:49:32 DEBUG { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG }, 10:49:32 DEBUG [...] 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG "data": { 10:49:32 DEBUG { 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "metric": { 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "result": [ 10:49:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:49:32 DEBUG 10:49:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:49:32 DEBUG { 10:49:32 DEBUG }, 10:49:32 DEBUG [...] 10:49:32 DEBUG }, 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG "value": [ 10:49:32 DEBUG "pod": "ds-idrepo-1" 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG }, 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "0" 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG "pod": "ds-cts-2" 10:49:32 DEBUG ] 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG }, 10:49:32 DEBUG } 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "value": [ 10:49:32 DEBUG ] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG } 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG "0.00429485169214678" 10:49:32 DEBUG } 10:49:32 DEBUG 10:49:32 DEBUG "pod": "ds-cts-2" 10:49:32 DEBUG ] 10:49:32 DEBUG 10:49:32 DEBUG }, 10:49:32 DEBUG } 10:49:32 DEBUG 10:49:32 DEBUG "value": [ 10:49:32 DEBUG ] 10:49:32 DEBUG "pod": "ds-idrepo-0" 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG } 10:49:32 DEBUG }, 10:49:32 DEBUG "802.8224225793807" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG } 10:49:32 DEBUG "value": [ 10:49:32 DEBUG ] 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG } 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG "747.8" 10:49:32 DEBUG ] 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG ] 10:49:32 DEBUG { 10:49:32 DEBUG } 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG } 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0.2975794126422529" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0.015" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG ] 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG } 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG ] 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG } 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:49:32 DEBUG } 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG 1713952172, 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG "0" 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG ] 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG } 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:49:32 DEBUG ] 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG } 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG } 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "ds-idrepo-1" 10:49:32 DEBUG "pod": "ds-idrepo-1" 10:49:32 DEBUG }, 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "ds-idrepo-0" 10:49:32 DEBUG "pod": "ds-idrepo-0" 10:49:32 DEBUG }, 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "747.8333333333334" 10:49:32 DEBUG "747.8666666666667" 10:49:32 DEBUG ] 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "instance": "10.142.0.24:9100" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "instance": "10.142.0.74:9100" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "2890804224" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952172" 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG { 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:49:32 DEBUG "data": { 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG { 10:49:32 DEBUG "result": [ 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG { 10:49:32 DEBUG "data": { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "pod": "ds-idrepo-1" 10:49:32 DEBUG "result": [ 10:49:32 DEBUG }, 10:49:32 DEBUG { 10:49:32 DEBUG "value": [ 10:49:32 DEBUG "metric": { 10:49:32 DEBUG 10:49:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:49:32 DEBUG 10:49:32 DEBUG }, 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952172" 10:49:32 DEBUG 10:49:32 DEBUG "value": [ 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG "pod": "ds-cts-0" 10:49:32 DEBUG 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:49:32 DEBUG }, 10:49:32 DEBUG 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG { 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "0" 10:49:32 DEBUG }, 10:49:32 DEBUG "data": { 10:49:32 DEBUG ] 10:49:32 DEBUG "value": [ 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG } 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "result": [ 10:49:32 DEBUG ] 10:49:32 DEBUG "1325096960" 10:49:32 DEBUG { 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG "metric": { 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 DEBUG "instance": "10.142.0.24:9100" 10:49:32 DEBUG ] 10:49:32 DEBUG }, 10:49:32 DEBUG } 10:49:32 DEBUG "value": [ 10:49:32 DEBUG } 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "instance": "10.142.0.74:9100" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "6457.4333333333325" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "pod": "ds-cts-1" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "pod": "ds-cts-0" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952172" 10:49:32 INFO [http_cmd]: http status code OK 10:49:32 DEBUG --- status code --- 10:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:49:32 DEBUG --- http response --- 10:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:49:32 DEBUG ----- output ----- 10:49:32 DEBUG { 10:49:32 DEBUG "status": "success", 10:49:32 DEBUG "data": { 10:49:32 DEBUG "resultType": "vector", 10:49:32 DEBUG "result": [ 10:49:32 DEBUG { 10:49:32 DEBUG "metric": { 10:49:32 DEBUG "instance": "10.142.0.24:9100" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG [...] 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG 10:49:32 DEBUG "instance": "10.142.0.74:9100" 10:49:32 DEBUG }, 10:49:32 DEBUG "value": [ 10:49:32 DEBUG 1713952172, 10:49:32 DEBUG "0" 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG ] 10:49:32 DEBUG } 10:49:32 DEBUG } 10:49:43 INFO 10:49:43 INFO [loop_until]: kubectl --namespace=xlou top pods 10:49:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:49:44 INFO [loop_until]: OK (rc = 0) 10:49:44 DEBUG --- stdout --- 10:49:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 3m 3466Mi am-769b98f8d5-2rr6f 4m 3408Mi am-769b98f8d5-h2r87 5m 3435Mi ds-cts-0 5m 559Mi ds-cts-1 6m 539Mi ds-cts-2 6m 503Mi ds-idrepo-0 4235m 11791Mi ds-idrepo-1 886m 11835Mi ds-idrepo-2 9m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 3890m 4824Mi idm-6c95d7588f-j6s6c 3790m 3312Mi lodemon-6b9df475cd-csxlf 5m 68Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 262m 1170Mi 10:49:44 DEBUG --- stderr --- 10:49:44 DEBUG 10:49:49 INFO 10:49:49 INFO [loop_until]: kubectl --namespace=xlou top node 10:49:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:49:49 INFO [loop_until]: OK (rc = 0) 10:49:49 DEBUG --- stdout --- 10:49:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 68m 0% 8243Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 638m 4% 13108Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4219m 26% 13103Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 59m 0% 1754Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 346m 2% 6152Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1722Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1638Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 81m 0% 4963Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4403m 27% 6375Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 138m 0% 2458Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 68m 0% 4830Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 61m 0% 1188Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 4163m 26% 4870Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 337m 2% 4602Mi 7% 10:49:49 DEBUG --- stderr --- 10:49:49 DEBUG 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "instance": "10.142.0.24:9100" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "instance": "10.142.0.74:9100" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "12080.799999999997" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "ds-cts-1" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "ds-cts-0" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "ds-cts-0" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "582.4389834068473" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 ERROR [http_cmd]: ERROR 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 ERROR --- status code --- 10:50:32 DEBUG --- status code --- 10:50:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 ERROR --- http response --- 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:50:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:50:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "ds-cts-0" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "ds-cts-1" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "ds-cts-0" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "instance": "10.142.0.24:9100" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "instance": "10.142.0.74:9100" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0.00020000000000000757" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "ds-idrepo-1" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "ds-idrepo-0" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "575.3333333333334" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "ds-cts-0" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "698.9267800882168" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG { 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "data": { 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "data": { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "ds-idrepo-1" 10:50:32 DEBUG "pod": "ds-idrepo-1" 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:50:32 DEBUG }, 10:50:32 DEBUG }, 10:50:32 DEBUG "result": [ 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG "value": [ 10:50:32 DEBUG { 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "metric": { 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG }, 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG [...] 10:50:32 DEBUG "value": [ 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "ds-idrepo-0" 10:50:32 DEBUG "pod": "ds-idrepo-0" 10:50:32 DEBUG [...] 10:50:32 DEBUG "pod": "ds-cts-0" 10:50:32 DEBUG }, 10:50:32 DEBUG }, 10:50:32 DEBUG 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "575.3333333333334" 10:50:32 DEBUG "575.2333333333333" 10:50:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:50:32 DEBUG "0.004384787229573524" 10:50:32 DEBUG ] 10:50:32 DEBUG ] 10:50:32 DEBUG }, 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 DEBUG "value": [ 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG ] 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 DEBUG "0" 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0.3227179721483907" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG } 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG } 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG { 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG "data": { 10:50:32 DEBUG { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG "data": { 10:50:32 DEBUG { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "result": [ 10:50:32 DEBUG "pod": "ds-idrepo-1" 10:50:32 DEBUG { 10:50:32 DEBUG }, 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "value": [ 10:50:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:50:32 DEBUG 10:50:32 DEBUG }, 10:50:32 DEBUG 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:50:32 DEBUG 10:50:32 DEBUG "pod": "ds-cts-0" 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG }, 10:50:32 DEBUG { 10:50:32 DEBUG 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG "value": [ 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "data": { 10:50:32 DEBUG }, 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG "0" 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "value": [ 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:50:32 DEBUG ] 10:50:32 DEBUG "result": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG } 10:50:32 DEBUG { 10:50:32 DEBUG "0.015" 10:50:32 DEBUG { 10:50:32 DEBUG ] 10:50:32 DEBUG "metric": { 10:50:32 DEBUG ] 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG } 10:50:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:50:32 DEBUG } 10:50:32 DEBUG "data": { 10:50:32 DEBUG } 10:50:32 DEBUG }, 10:50:32 DEBUG ] 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "value": [ 10:50:32 DEBUG } 10:50:32 DEBUG "result": [ 10:50:32 DEBUG 10:50:32 DEBUG } 10:50:32 DEBUG { 10:50:32 DEBUG 10:50:32 DEBUG "metric": { 10:50:32 DEBUG 10:50:32 DEBUG "instance": "10.142.0.24:9100" 10:50:32 DEBUG [...] 10:50:32 DEBUG }, 10:50:32 DEBUG 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:50:32 DEBUG 10:50:32 DEBUG }, 10:50:32 DEBUG [...] 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG 10:50:32 DEBUG "0" 10:50:32 DEBUG 10:50:32 DEBUG ] 10:50:32 DEBUG "instance": "10.142.0.74:9100" 10:50:32 DEBUG } 10:50:32 DEBUG }, 10:50:32 DEBUG ] 10:50:32 DEBUG "value": [ 10:50:32 DEBUG } 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG } 10:50:32 DEBUG "2891107328" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952232" 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "instance": "10.142.0.24:9100" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "instance": "10.142.0.74:9100" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "7140.466666666666" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "1320706048" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "pod": "ds-cts-1" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "pod": "ds-cts-0" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952232" 10:50:32 INFO [http_cmd]: http status code OK 10:50:32 DEBUG --- status code --- 10:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:50:32 DEBUG --- http response --- 10:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:50:32 DEBUG ----- output ----- 10:50:32 DEBUG { 10:50:32 DEBUG "status": "success", 10:50:32 DEBUG "data": { 10:50:32 DEBUG "resultType": "vector", 10:50:32 DEBUG "result": [ 10:50:32 DEBUG { 10:50:32 DEBUG "metric": { 10:50:32 DEBUG "instance": "10.142.0.24:9100" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG [...] 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG 10:50:32 DEBUG "instance": "10.142.0.74:9100" 10:50:32 DEBUG }, 10:50:32 DEBUG "value": [ 10:50:32 DEBUG 1713952232, 10:50:32 DEBUG "0" 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG ] 10:50:32 DEBUG } 10:50:32 DEBUG } 10:50:44 INFO 10:50:44 INFO [loop_until]: kubectl --namespace=xlou top pods 10:50:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:50:44 INFO [loop_until]: OK (rc = 0) 10:50:44 DEBUG --- stdout --- 10:50:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 4m 3408Mi am-769b98f8d5-h2r87 5m 3435Mi ds-cts-0 5m 559Mi ds-cts-1 7m 539Mi ds-cts-2 5m 503Mi ds-idrepo-0 4444m 11790Mi ds-idrepo-1 375m 11835Mi ds-idrepo-2 11m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4655m 4594Mi idm-6c95d7588f-j6s6c 3575m 3314Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 263m 1170Mi 10:50:44 DEBUG --- stderr --- 10:50:44 DEBUG 10:50:49 INFO 10:50:49 INFO [loop_until]: kubectl --namespace=xlou top node 10:50:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:50:49 INFO [loop_until]: OK (rc = 0) 10:50:49 DEBUG --- stdout --- 10:50:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 68m 0% 8237Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 560m 3% 13111Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4185m 26% 13104Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 64m 0% 1751Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 356m 2% 6131Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 57m 0% 1731Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 55m 0% 1638Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 80m 0% 4981Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4737m 29% 6380Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 151m 0% 2459Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 67m 0% 4839Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 60m 0% 1190Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 4081m 25% 4883Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 348m 2% 4598Mi 7% 10:50:49 DEBUG --- stderr --- 10:50:49 DEBUG 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "instance": "10.142.0.24:9100" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "instance": "10.142.0.74:9100" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "13446.466666666667" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "pod": "ds-cts-1" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "pod": "ds-cts-0" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "0" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "pod": "ds-cts-0" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "10657.997429449568" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "0" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952292" 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 ERROR [http_cmd]: ERROR 10:51:32 ERROR --- status code --- 10:51:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:51:32 ERROR --- http response --- 10:51:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:51:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "pod": "ds-cts-0" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "0" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:51:32 DEBUG { 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG { 10:51:32 DEBUG { 10:51:32 DEBUG "data": { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "data": { 10:51:32 DEBUG "result": [ 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG { 10:51:32 DEBUG "result": [ 10:51:32 DEBUG "result": [ 10:51:32 DEBUG "metric": { 10:51:32 DEBUG { 10:51:32 DEBUG { 10:51:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG }, 10:51:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:51:32 DEBUG "pod": "ds-idrepo-1" 10:51:32 DEBUG "value": [ 10:51:32 DEBUG }, 10:51:32 DEBUG }, 10:51:32 DEBUG 10:51:32 DEBUG "value": [ 10:51:32 DEBUG "value": [ 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG [...] 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952292" 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG }, 10:51:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:51:32 DEBUG "pod": "ds-idrepo-0" 10:51:32 DEBUG "value": [ 10:51:32 DEBUG }, 10:51:32 DEBUG }, 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG "value": [ 10:51:32 DEBUG "0" 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG ] 10:51:32 DEBUG "0" 10:51:32 DEBUG "544.5333333333333" 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG ] 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "pod": "ds-cts-1" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "pod": "ds-cts-0" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "0" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "instance": "10.142.0.24:9100" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "instance": "10.142.0.74:9100" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "0.0002333333333332632" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:51:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:51:32 DEBUG }, 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "pod": "ds-cts-0" 10:51:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:51:32 DEBUG }, 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "479.01112042469964" 10:51:32 DEBUG "0" 10:51:32 DEBUG ] 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG { 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:51:32 DEBUG "data": { 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "result": [ 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG { 10:51:32 DEBUG "data": { 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG { 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "data": { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG "metric": { 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "pod": "ds-idrepo-1" 10:51:32 DEBUG "result": [ 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG "pod": "ds-idrepo-1" 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG }, 10:51:32 DEBUG "result": [ 10:51:32 DEBUG }, 10:51:32 DEBUG { 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG }, 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG "value": [ 10:51:32 DEBUG { 10:51:32 DEBUG "value": [ 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "value": [ 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:51:32 DEBUG 10:51:32 DEBUG "metric": { 10:51:32 DEBUG 10:51:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:51:32 DEBUG 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG 10:51:32 DEBUG "instance": "10.142.0.24:9100" 10:51:32 DEBUG 10:51:32 DEBUG }, 10:51:32 DEBUG 10:51:32 DEBUG { 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG { 10:51:32 DEBUG 10:51:32 DEBUG }, 10:51:32 DEBUG 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG "status": "success", 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952292" 10:51:32 DEBUG [...] 10:51:32 DEBUG "value": [ 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG "data": { 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:51:32 DEBUG "data": { 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "result": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG { 10:51:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:51:32 DEBUG [...] 10:51:32 DEBUG "pod": "ds-cts-0" 10:51:32 DEBUG 10:51:32 DEBUG "pod": "ds-idrepo-0" 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "data": { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG }, 10:51:32 DEBUG 10:51:32 DEBUG }, 10:51:32 DEBUG 10:51:32 DEBUG }, 10:51:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "pod": "ds-idrepo-1" 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG "value": [ 10:51:32 DEBUG }, 10:51:32 DEBUG "result": [ 10:51:32 DEBUG }, 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG { 10:51:32 DEBUG "value": [ 10:51:32 DEBUG "0.015" 10:51:32 DEBUG "instance": "10.142.0.74:9100" 10:51:32 DEBUG "0" 10:51:32 DEBUG }, 10:51:32 DEBUG "544.5333333333333" 10:51:32 DEBUG 10:51:32 DEBUG "metric": { 10:51:32 DEBUG 10:51:32 DEBUG ] 10:51:32 DEBUG }, 10:51:32 DEBUG ] 10:51:32 DEBUG "value": [ 10:51:32 DEBUG ] 10:51:32 DEBUG 10:51:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:51:32 DEBUG 10:51:32 DEBUG } 10:51:32 DEBUG "value": [ 10:51:32 DEBUG } 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG } 10:51:32 DEBUG 10:51:32 DEBUG }, 10:51:32 DEBUG 10:51:32 DEBUG ] 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG ] 10:51:32 DEBUG "0" 10:51:32 DEBUG ] 10:51:32 DEBUG [...] 10:51:32 DEBUG "value": [ 10:51:32 DEBUG [...] 10:51:32 DEBUG } 10:51:32 DEBUG "2891330560" 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG "pod": "ds-cts-0" 10:51:32 DEBUG [...] 10:51:32 DEBUG "pod": "ds-idrepo-0" 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG }, 10:51:32 DEBUG 10:51:32 DEBUG }, 10:51:32 DEBUG } 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG "value": [ 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG "0.0014984300502935296" 10:51:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:51:32 DEBUG "544.5333333333333" 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:51:32 DEBUG ] 10:51:32 DEBUG }, 10:51:32 DEBUG ] 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG } 10:51:32 DEBUG "value": [ 10:51:32 DEBUG } 10:51:32 DEBUG { 10:51:32 DEBUG ] 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG ] 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG } 10:51:32 DEBUG "1322876928" 10:51:32 DEBUG } 10:51:32 DEBUG "data": { 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG } 10:51:32 DEBUG "result": [ 10:51:32 DEBUG ] 10:51:32 DEBUG { 10:51:32 DEBUG } 10:51:32 DEBUG "metric": { 10:51:32 DEBUG } 10:51:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "0.31691198410665494" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "0" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952292" 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952292" 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "instance": "10.142.0.24:9100" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "instance": "10.142.0.74:9100" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "8326.766666666666" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "pod": "ds-cts-1" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "pod": "ds-cts-0" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "0" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952292" 10:51:32 INFO [http_cmd]: http status code OK 10:51:32 DEBUG --- status code --- 10:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:51:32 DEBUG --- http response --- 10:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:51:32 DEBUG ----- output ----- 10:51:32 DEBUG { 10:51:32 DEBUG "status": "success", 10:51:32 DEBUG "data": { 10:51:32 DEBUG "resultType": "vector", 10:51:32 DEBUG "result": [ 10:51:32 DEBUG { 10:51:32 DEBUG "metric": { 10:51:32 DEBUG "instance": "10.142.0.24:9100" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG [...] 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG 10:51:32 DEBUG "instance": "10.142.0.74:9100" 10:51:32 DEBUG }, 10:51:32 DEBUG "value": [ 10:51:32 DEBUG 1713952292, 10:51:32 DEBUG "0.00033333333333333365" 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG ] 10:51:32 DEBUG } 10:51:32 DEBUG } 10:51:44 INFO 10:51:44 INFO [loop_until]: kubectl --namespace=xlou top pods 10:51:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:51:44 INFO [loop_until]: OK (rc = 0) 10:51:44 DEBUG --- stdout --- 10:51:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 4m 3408Mi am-769b98f8d5-h2r87 5m 3435Mi ds-cts-0 5m 559Mi ds-cts-1 6m 539Mi ds-cts-2 5m 503Mi ds-idrepo-0 4190m 11790Mi ds-idrepo-1 20m 11836Mi ds-idrepo-2 13m 5820Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 4048m 4594Mi idm-6c95d7588f-j6s6c 3102m 3331Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 259m 1170Mi 10:51:44 DEBUG --- stderr --- 10:51:44 DEBUG 10:51:49 INFO 10:51:49 INFO [loop_until]: kubectl --namespace=xlou top node 10:51:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:51:49 INFO [loop_until]: OK (rc = 0) 10:51:49 DEBUG --- stdout --- 10:51:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 70m 0% 8250Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 64m 0% 13098Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4428m 27% 13103Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 59m 0% 1762Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 338m 2% 6140Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 58m 0% 1721Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1636Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 81m 0% 4983Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4122m 25% 6154Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 142m 0% 2471Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 68m 0% 4828Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 59m 0% 1186Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 3198m 20% 4886Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 326m 2% 4595Mi 7% 10:51:49 DEBUG --- stderr --- 10:51:49 DEBUG 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "instance": "10.142.0.24:9100" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "instance": "10.142.0.74:9100" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "12933.633333333333" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "pod": "ds-cts-1" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "pod": "ds-cts-0" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "0" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "0" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "0" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952352" 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:52:32 ERROR [http_cmd]: ERROR 10:52:32 ERROR --- status code --- 10:52:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:52:32 ERROR --- http response --- 10:52:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:52:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "pod": "ds-cts-0" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "0" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952352" 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG --- status code --- 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG { 10:52:32 DEBUG { 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "data": { 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "result": [ 10:52:32 DEBUG "result": [ 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG { 10:52:32 DEBUG { 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "pod": "ds-idrepo-1" 10:52:32 DEBUG }, 10:52:32 DEBUG "instance": "10.142.0.24:9100" 10:52:32 DEBUG "pod": "ds-cts-1" 10:52:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG }, 10:52:32 DEBUG }, 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG "value": [ 10:52:32 DEBUG "value": [ 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG [...] 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "pod": "ds-idrepo-0" 10:52:32 DEBUG }, 10:52:32 DEBUG "instance": "10.142.0.74:9100" 10:52:32 DEBUG "pod": "ds-cts-0" 10:52:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG }, 10:52:32 DEBUG }, 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG "value": [ 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "0" 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "626.3333333333334" 10:52:32 DEBUG ] 10:52:32 DEBUG "0.0003333333333333854" 10:52:32 DEBUG "0" 10:52:32 DEBUG "0" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG ] 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG ] 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "0" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "0.27963135848254944" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952352" 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "pod": "ds-cts-0" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "453.549846558527" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "1322688512" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952352" 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 DEBUG --- http response --- 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "0" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952352" 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG "metric": { 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG "instance": "10.142.0.24:9100" 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG }, 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG "value": [ 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:52:32 DEBUG 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952352" 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG { 10:52:32 DEBUG 10:52:32 DEBUG { 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG [...] 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG "data": { 10:52:32 DEBUG 10:52:32 DEBUG "data": { 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG "result": [ 10:52:32 DEBUG 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG { 10:52:32 DEBUG { 10:52:32 DEBUG { 10:52:32 DEBUG "instance": "10.142.0.74:9100" 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "metric": { 10:52:32 DEBUG }, 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "data": { 10:52:32 DEBUG "data": { 10:52:32 DEBUG "data": { 10:52:32 DEBUG "pod": "ds-idrepo-1" 10:52:32 DEBUG "value": [ 10:52:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "resultType": "vector", 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG }, 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG }, 10:52:32 DEBUG "result": [ 10:52:32 DEBUG "result": [ 10:52:32 DEBUG "result": [ 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG "value": [ 10:52:32 DEBUG "7527.7666666666655" 10:52:32 DEBUG "value": [ 10:52:32 DEBUG { 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG 10:52:32 DEBUG ] 10:52:32 DEBUG 10:52:32 DEBUG "metric": { 10:52:32 DEBUG { 10:52:32 DEBUG "pod": "ds-idrepo-1" 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG 10:52:32 DEBUG } 10:52:32 DEBUG 10:52:32 DEBUG "pod": "ds-idrepo-1" 10:52:32 DEBUG "metric": { 10:52:32 DEBUG }, 10:52:32 DEBUG 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG ] 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG 10:52:32 DEBUG }, 10:52:32 DEBUG "instance": "10.142.0.24:9100" 10:52:32 DEBUG "value": [ 10:52:32 DEBUG [...] 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG } 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG [...] 10:52:32 DEBUG "value": [ 10:52:32 DEBUG }, 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG } 10:52:32 DEBUG { 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:52:32 DEBUG "data": { 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG "pod": "ds-idrepo-0" 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG }, 10:52:32 DEBUG { 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG }, 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG "value": [ 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "data": { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "data": { 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "pod": "ds-cts-0" 10:52:32 DEBUG "626.3666666666667" 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "0.015" 10:52:32 DEBUG "pod": "ds-idrepo-0" 10:52:32 DEBUG 10:52:32 DEBUG }, 10:52:32 DEBUG ] 10:52:32 DEBUG "result": [ 10:52:32 DEBUG }, 10:52:32 DEBUG "result": [ 10:52:32 DEBUG ] 10:52:32 DEBUG }, 10:52:32 DEBUG "instance": "10.142.0.74:9100" 10:52:32 DEBUG "value": [ 10:52:32 DEBUG } 10:52:32 DEBUG { 10:52:32 DEBUG "value": [ 10:52:32 DEBUG { 10:52:32 DEBUG } 10:52:32 DEBUG "value": [ 10:52:32 DEBUG }, 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG ] 10:52:32 DEBUG "metric": { 10:52:32 DEBUG 10:52:32 DEBUG "metric": { 10:52:32 DEBUG ] 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG "0" 10:52:32 DEBUG } 10:52:32 DEBUG "pod": "ds-cts-1" 10:52:32 DEBUG 10:52:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:52:32 DEBUG } 10:52:32 DEBUG "626.3" 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG }, 10:52:32 DEBUG 10:52:32 DEBUG }, 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG "2891625472" 10:52:32 DEBUG } 10:52:32 DEBUG "value": [ 10:52:32 DEBUG [...] 10:52:32 DEBUG "value": [ 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG ] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG [...] 10:52:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:52:32 DEBUG [...] 10:52:32 DEBUG } 10:52:32 DEBUG 10:52:32 DEBUG }, 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG 10:52:32 DEBUG "pod": "ds-cts-0" 10:52:32 DEBUG "0" 10:52:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:52:32 DEBUG }, 10:52:32 DEBUG ] 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG } 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG ] 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "0" 10:52:32 DEBUG } 10:52:32 DEBUG "0" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952352" 10:52:32 INFO [http_cmd]: http status code OK 10:52:32 DEBUG --- status code --- 10:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:52:32 DEBUG --- http response --- 10:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:52:32 DEBUG ----- output ----- 10:52:32 DEBUG { 10:52:32 DEBUG "status": "success", 10:52:32 DEBUG "data": { 10:52:32 DEBUG "resultType": "vector", 10:52:32 DEBUG "result": [ 10:52:32 DEBUG { 10:52:32 DEBUG "metric": { 10:52:32 DEBUG "instance": "10.142.0.24:9100" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG [...] 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG 10:52:32 DEBUG "instance": "10.142.0.74:9100" 10:52:32 DEBUG }, 10:52:32 DEBUG "value": [ 10:52:32 DEBUG 1713952352, 10:52:32 DEBUG "0" 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG ] 10:52:32 DEBUG } 10:52:32 DEBUG } 10:52:44 INFO 10:52:44 INFO [loop_until]: kubectl --namespace=xlou top pods 10:52:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:52:44 INFO [loop_until]: OK (rc = 0) 10:52:44 DEBUG --- stdout --- 10:52:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 4m 3409Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 5m 559Mi ds-cts-1 6m 539Mi ds-cts-2 5m 503Mi ds-idrepo-0 1376m 11788Mi ds-idrepo-1 1893m 11832Mi ds-idrepo-2 8m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 5262m 4592Mi idm-6c95d7588f-j6s6c 2496m 3389Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 189m 145Mi 10:52:44 DEBUG --- stderr --- 10:52:44 DEBUG 10:52:49 INFO 10:52:49 INFO [loop_until]: kubectl --namespace=xlou top node 10:52:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:52:49 INFO [loop_until]: OK (rc = 0) 10:52:49 DEBUG --- stdout --- 10:52:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 68m 0% 8251Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 2154m 13% 13108Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 1161m 7% 13082Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 60m 0% 1754Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 218m 1% 6136Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 57m 0% 1724Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 55m 0% 1641Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 82m 0% 4978Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 4908m 30% 6158Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 146m 0% 2447Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 65m 0% 4821Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 58m 0% 1188Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 2413m 15% 4942Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 315m 1% 3582Mi 6% 10:52:49 DEBUG --- stderr --- 10:52:49 DEBUG 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "instance": "10.142.0.24:9100" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "instance": "10.142.0.74:9100" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "12229.666666666668" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "ds-cts-1" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "ds-cts-0" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 ERROR [http_cmd]: ERROR 10:53:32 ERROR --- status code --- 10:53:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:53:32 ERROR --- http response --- 10:53:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:53:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "ds-cts-1" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "ds-cts-0" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "ds-idrepo-1" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "ds-idrepo-0" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG } 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG ] 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG } 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG } 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "instance": "10.142.0.24:9100" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG "instance": "10.142.0.74:9100" 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG }, 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG "value": [ 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG { 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "0.00020000000000000757" 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG "data": { 10:53:32 DEBUG ] 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:53:32 DEBUG } 10:53:32 DEBUG "result": [ 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG ] 10:53:32 DEBUG { 10:53:32 DEBUG { 10:53:32 DEBUG } 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG } 10:53:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:53:32 DEBUG "data": { 10:53:32 DEBUG }, 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "value": [ 10:53:32 DEBUG "result": [ 10:53:32 DEBUG 10:53:32 DEBUG { 10:53:32 DEBUG 10:53:32 DEBUG "metric": { 10:53:32 DEBUG 10:53:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:53:32 DEBUG [...] 10:53:32 DEBUG }, 10:53:32 DEBUG 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:53:32 DEBUG 10:53:32 DEBUG }, 10:53:32 DEBUG [...] 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG 10:53:32 DEBUG "249090048" 10:53:32 DEBUG 10:53:32 DEBUG ] 10:53:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:53:32 DEBUG } 10:53:32 DEBUG }, 10:53:32 DEBUG ] 10:53:32 DEBUG "value": [ 10:53:32 DEBUG } 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG } 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952412" 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0.07144665150120463" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "instance": "10.142.0.24:9100" 10:53:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:53:32 DEBUG }, 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG "instance": "10.142.0.74:9100" 10:53:32 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG }, 10:53:32 DEBUG }, 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG "value": [ 10:53:32 DEBUG "value": [ 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG "2891906048" 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG "0.000599391249760236" 10:53:32 DEBUG { 10:53:32 DEBUG ] 10:53:32 DEBUG { 10:53:32 DEBUG ] 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG } 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG } 10:53:32 DEBUG "data": { 10:53:32 DEBUG ] 10:53:32 DEBUG "data": { 10:53:32 DEBUG ] 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG } 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG } 10:53:32 DEBUG "result": [ 10:53:32 DEBUG } 10:53:32 DEBUG "result": [ 10:53:32 DEBUG } 10:53:32 DEBUG { 10:53:32 DEBUG { 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "ds-cts-1" 10:53:32 DEBUG "pod": "ds-idrepo-1" 10:53:32 DEBUG }, 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "ds-cts-0" 10:53:32 DEBUG "pod": "ds-idrepo-0" 10:53:32 DEBUG }, 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 DEBUG { 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "pod": "ds-idrepo-1" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "ds-idrepo-0" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952412" 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG { 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "data": { 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "result": [ 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG { 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG "metric": { 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG "instance": "10.142.0.24:9100" 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG }, 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:53:32 DEBUG "value": [ 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG 10:53:32 DEBUG { 10:53:32 DEBUG 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG 10:53:32 DEBUG "data": { 10:53:32 DEBUG [...] 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG 10:53:32 DEBUG "result": [ 10:53:32 DEBUG 10:53:32 DEBUG { 10:53:32 DEBUG 10:53:32 DEBUG "metric": { 10:53:32 DEBUG "instance": "10.142.0.74:9100" 10:53:32 DEBUG "instance": "10.142.0.24:9100" 10:53:32 DEBUG }, 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG 10:53:32 DEBUG "7021.066666666667" 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG 10:53:32 DEBUG ] 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:53:32 DEBUG 10:53:32 DEBUG } 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG [...] 10:53:32 DEBUG ] 10:53:32 DEBUG { 10:53:32 DEBUG 10:53:32 DEBUG } 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG 10:53:32 DEBUG } 10:53:32 DEBUG "data": { 10:53:32 DEBUG 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "instance": "10.142.0.74:9100" 10:53:32 DEBUG "result": [ 10:53:32 DEBUG }, 10:53:32 DEBUG { 10:53:32 DEBUG "value": [ 10:53:32 DEBUG "metric": { 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "pod": "ds-idrepo-1" 10:53:32 DEBUG "0" 10:53:32 DEBUG }, 10:53:32 DEBUG ] 10:53:32 DEBUG "value": [ 10:53:32 INFO [http_cmd]: http status code OK 10:53:32 DEBUG } 10:53:32 DEBUG 10:53:32 DEBUG --- status code --- 10:53:32 DEBUG ] 10:53:32 DEBUG 10:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:53:32 DEBUG } 10:53:32 DEBUG 10:53:32 DEBUG --- http response --- 10:53:32 DEBUG } 10:53:32 DEBUG [...] 10:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:53:32 DEBUG 10:53:32 DEBUG ----- output ----- 10:53:32 DEBUG 10:53:32 DEBUG { 10:53:32 DEBUG 10:53:32 DEBUG "status": "success", 10:53:32 DEBUG "pod": "ds-cts-0" 10:53:32 DEBUG "data": { 10:53:32 DEBUG }, 10:53:32 DEBUG "resultType": "vector", 10:53:32 DEBUG "value": [ 10:53:32 DEBUG "result": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG { 10:53:32 DEBUG "0" 10:53:32 DEBUG "metric": { 10:53:32 DEBUG ] 10:53:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:53:32 DEBUG } 10:53:32 DEBUG }, 10:53:32 DEBUG ] 10:53:32 DEBUG "value": [ 10:53:32 DEBUG } 10:53:32 DEBUG 10:53:32 DEBUG } 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG [...] 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG 10:53:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:53:32 DEBUG }, 10:53:32 DEBUG "value": [ 10:53:32 DEBUG 1713952412, 10:53:32 DEBUG "0.015" 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG ] 10:53:32 DEBUG } 10:53:32 DEBUG } 10:53:44 INFO 10:53:44 INFO [loop_until]: kubectl --namespace=xlou top pods 10:53:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:53:44 INFO [loop_until]: OK (rc = 0) 10:53:44 DEBUG --- stdout --- 10:53:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 3m 3409Mi am-769b98f8d5-h2r87 3m 3435Mi ds-cts-0 5m 559Mi ds-cts-1 5m 540Mi ds-cts-2 5m 503Mi ds-idrepo-0 17m 11788Mi ds-idrepo-1 9m 11832Mi ds-idrepo-2 9m 5819Mi end-user-ui-58878d7668-zkqbp 1m 3Mi idm-6c95d7588f-98n7b 5m 4591Mi idm-6c95d7588f-j6s6c 14m 3385Mi lodemon-6b9df475cd-csxlf 2m 67Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 1m 145Mi 10:53:44 DEBUG --- stderr --- 10:53:44 DEBUG 10:53:49 INFO 10:53:49 INFO [loop_until]: kubectl --namespace=xlou top node 10:53:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:53:49 INFO [loop_until]: OK (rc = 0) 10:53:49 DEBUG --- stdout --- 10:53:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 67m 0% 8250Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 58m 0% 13102Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 70m 0% 13081Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 55m 0% 1755Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 62m 0% 6123Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1735Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 57m 0% 1641Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 80m 0% 4969Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 67m 0% 6140Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 151m 0% 2456Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 61m 0% 4837Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 60m 0% 1192Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 76m 0% 4939Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 65m 0% 3580Mi 6% 10:53:49 DEBUG --- stderr --- 10:53:49 DEBUG 10:54:18 WARNING File /tmp/lodemon_data-20240424_062037/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [24/Apr/2024 10:54:18] "GET /monitoring/average?start_time=20240424_092342&stop_time=20240424_095217 HTTP/1.1" 200 - 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "instance": "10.142.0.24:9100" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "instance": "10.142.0.74:9100" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "12639.133333333333" 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "pod": "ds-cts-0" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0" 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "ds-cts-1" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "pod": "ds-cts-0" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0" 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0" 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0" 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 ERROR [http_cmd]: ERROR 10:54:32 ERROR --- status code --- 10:54:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:54:32 ERROR --- http response --- 10:54:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:54:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "ds-idrepo-1" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "pod": "ds-idrepo-0" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0" 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:54:32 DEBUG "pod": "ds-cts-1" 10:54:32 DEBUG }, 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:54:32 DEBUG "pod": "ds-cts-0" 10:54:32 DEBUG }, 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0" 10:54:32 DEBUG "0" 10:54:32 DEBUG ] 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0" 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG { 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG "data": { 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "data": { 10:54:32 DEBUG "instance": "10.142.0.24:9100" 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG }, 10:54:32 DEBUG "result": [ 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 DEBUG "value": [ 10:54:32 DEBUG { 10:54:32 DEBUG 10:54:32 DEBUG "metric": { 10:54:32 DEBUG 10:54:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:54:32 DEBUG }, 10:54:32 DEBUG 10:54:32 DEBUG "value": [ 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG "instance": "10.142.0.74:9100" 10:54:32 DEBUG 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG }, 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG "value": [ 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG "0.0001666666666666335" 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG }, 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:54:32 DEBUG ] 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:54:32 DEBUG "value": [ 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG } 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG ] 10:54:32 DEBUG { 10:54:32 DEBUG { 10:54:32 DEBUG "0.02696068013974851" 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG } 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG ] 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG } 10:54:32 DEBUG "data": { 10:54:32 DEBUG "data": { 10:54:32 DEBUG } 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG ] 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG "result": [ 10:54:32 DEBUG "result": [ 10:54:32 DEBUG } 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG { 10:54:32 DEBUG { 10:54:32 DEBUG } 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "ds-idrepo-1" 10:54:32 DEBUG "instance": "10.142.0.24:9100" 10:54:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:54:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:54:32 DEBUG }, 10:54:32 DEBUG }, 10:54:32 DEBUG }, 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG [...] 10:54:32 DEBUG [...] 10:54:32 DEBUG [...] 10:54:32 DEBUG [...] 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG "pod": "ds-idrepo-0" 10:54:32 DEBUG "instance": "10.142.0.74:9100" 10:54:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:54:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:54:32 DEBUG { 10:54:32 DEBUG }, 10:54:32 DEBUG }, 10:54:32 DEBUG }, 10:54:32 DEBUG }, 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "data": { 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "0" 10:54:32 DEBUG "2892127232" 10:54:32 DEBUG "0" 10:54:32 DEBUG "251539456" 10:54:32 DEBUG "result": [ 10:54:32 DEBUG ] 10:54:32 DEBUG ] 10:54:32 DEBUG ] 10:54:32 DEBUG ] 10:54:32 DEBUG { 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG "metric": { 10:54:32 DEBUG ] 10:54:32 DEBUG ] 10:54:32 DEBUG ] 10:54:32 DEBUG ] 10:54:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG }, 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "pod": "overseer-0-55d5975bd7-gp92m" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0" 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "ds-idrepo-1" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "pod": "ds-idrepo-0" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0" 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952472" 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG { 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG "data": { 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "data": { 10:54:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG }, 10:54:32 DEBUG "result": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG { 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG 10:54:32 DEBUG "metric": { 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG 10:54:32 DEBUG "pod": "ds-cts-1" 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG 10:54:32 DEBUG }, 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [...] 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG { 10:54:32 DEBUG 10:54:32 DEBUG "pod": "ds-cts-0" 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG [...] 10:54:32 DEBUG }, 10:54:32 DEBUG "data": { 10:54:32 DEBUG 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "result": [ 10:54:32 DEBUG 10:54:32 DEBUG "0" 10:54:32 DEBUG { 10:54:32 DEBUG "pod": "ds-cts-0" 10:54:32 DEBUG ] 10:54:32 DEBUG "metric": { 10:54:32 DEBUG }, 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:54:32 DEBUG "value": [ 10:54:32 DEBUG ] 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG }, 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG } 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "0" 10:54:32 DEBUG } 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG 10:54:32 DEBUG ] 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:54:32 DEBUG 10:54:32 DEBUG } 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG 10:54:32 DEBUG ] 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG [...] 10:54:32 DEBUG } 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG 10:54:32 DEBUG } 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG }, 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "instance": "10.142.0.24:9100" 10:54:32 DEBUG }, 10:54:32 DEBUG "0" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG ] 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG } 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG ] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG } 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG } 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:54:32 DEBUG "instance": "10.142.0.74:9100" 10:54:32 DEBUG }, 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0" 10:54:32 DEBUG "7427.599999999999" 10:54:32 DEBUG ] 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952472" 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG { 10:54:32 DEBUG { 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG "data": { 10:54:32 DEBUG "data": { 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG "result": [ 10:54:32 DEBUG "result": [ 10:54:32 DEBUG { 10:54:32 DEBUG { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "metric": { 10:54:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:54:32 DEBUG "pod": "ds-idrepo-1" 10:54:32 DEBUG }, 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 INFO [http_cmd]: http status code OK 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG --- status code --- 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG --- http response --- 10:54:32 DEBUG 10:54:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:54:32 DEBUG "pod": "ds-cts-0" 10:54:32 DEBUG }, 10:54:32 DEBUG ----- output ----- 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG { 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "status": "success", 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0.015" 10:54:32 DEBUG "data": { 10:54:32 DEBUG "0" 10:54:32 DEBUG ] 10:54:32 DEBUG "resultType": "vector", 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG "result": [ 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG { 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG "metric": { 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:32 DEBUG "instance": "10.142.0.24:9100" 10:54:32 DEBUG } 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG [...] 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG 10:54:32 DEBUG "instance": "10.142.0.74:9100" 10:54:32 DEBUG }, 10:54:32 DEBUG "value": [ 10:54:32 DEBUG 1713952472, 10:54:32 DEBUG "0.00033333333333333365" 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG ] 10:54:32 DEBUG } 10:54:32 DEBUG } 10:54:44 INFO 10:54:44 INFO [loop_until]: kubectl --namespace=xlou top pods 10:54:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:54:44 INFO [loop_until]: OK (rc = 0) 10:54:44 DEBUG --- stdout --- 10:54:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 6Mi am-769b98f8d5-2pqrk 4m 3466Mi am-769b98f8d5-2rr6f 3m 3409Mi am-769b98f8d5-h2r87 4m 3435Mi ds-cts-0 4m 559Mi ds-cts-1 5m 540Mi ds-cts-2 10m 503Mi ds-idrepo-0 12m 11788Mi ds-idrepo-1 14m 11833Mi ds-idrepo-2 15m 5819Mi end-user-ui-58878d7668-zkqbp 1m 4Mi idm-6c95d7588f-98n7b 8m 4590Mi idm-6c95d7588f-j6s6c 5m 3385Mi lodemon-6b9df475cd-csxlf 5m 68Mi login-ui-5fdf49fc66-7c9cd 1m 3Mi overseer-0-55d5975bd7-gp92m 8m 145Mi 10:54:44 DEBUG --- stderr --- 10:54:44 DEBUG 10:54:49 INFO 10:54:49 INFO [loop_until]: kubectl --namespace=xlou top node 10:54:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:54:49 INFO [loop_until]: OK (rc = 0) 10:54:49 DEBUG --- stdout --- 10:54:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 75m 0% 8250Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 58m 0% 13101Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 65m 0% 13079Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 59m 0% 1747Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 66m 0% 6114Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 58m 0% 1727Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 62m 0% 1632Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 83m 0% 4974Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 72m 0% 6137Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 148m 0% 2462Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 62m 0% 4831Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 61m 0% 1194Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 66m 0% 4922Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 99m 0% 3594Mi 6% 10:54:49 DEBUG --- stderr --- 10:54:49 DEBUG 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "instance": "10.142.0.24:9100" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "instance": "10.142.0.74:9100" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "14881.066666666666" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "ds-idrepo-1" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "39457.87935158119" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "ds-cts-1" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "ds-cts-0" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 ERROR [http_cmd]: ERROR 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 ERROR --- status code --- 10:55:32 DEBUG --- status code --- 10:55:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 ERROR --- http response --- 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:55:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "ds-idrepo-1" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "ds-idrepo-0" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "2.6666666666666665" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "ds-idrepo-1" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "ds-cts-1" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "ds-cts-0" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "ds-idrepo-1" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "ds-idrepo-1" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "ds-idrepo-0" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "2.6666666666666665" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:55:32 DEBUG "instance": "10.142.0.24:9100" 10:55:32 DEBUG }, 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:55:32 DEBUG 10:55:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG "instance": "10.142.0.74:9100" 10:55:32 DEBUG }, 10:55:32 DEBUG { 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "data": { 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0.031753284668142416" 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "0.0004333333333332708" 10:55:32 DEBUG ] 10:55:32 DEBUG "result": [ 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG { 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG "metric": { 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG "instance": "10.142.0.24:9100" 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG }, 10:55:32 DEBUG } 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG "value": [ 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG 10:55:32 DEBUG --- http response --- 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG [...] 10:55:32 DEBUG { 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:55:32 DEBUG 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG "data": { 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG { 10:55:32 DEBUG "instance": "10.142.0.74:9100" 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG "result": [ 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG }, 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG { 10:55:32 DEBUG "data": { 10:55:32 DEBUG "value": [ 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:55:32 DEBUG "result": [ 10:55:32 DEBUG "2892788736" 10:55:32 DEBUG { 10:55:32 DEBUG }, 10:55:32 DEBUG { 10:55:32 DEBUG ] 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "value": [ 10:55:32 DEBUG "metric": { 10:55:32 DEBUG } 10:55:32 DEBUG "data": { 10:55:32 DEBUG 10:55:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:55:32 DEBUG ] 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG 10:55:32 DEBUG }, 10:55:32 DEBUG } 10:55:32 DEBUG "result": [ 10:55:32 DEBUG 10:55:32 DEBUG "value": [ 10:55:32 DEBUG } 10:55:32 DEBUG { 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG "metric": { 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "ds-idrepo-1" 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG }, 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG "value": [ 10:55:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG }, 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:55:32 DEBUG [...] 10:55:32 DEBUG "0.06666666666666667" 10:55:32 DEBUG }, 10:55:32 DEBUG 10:55:32 DEBUG ] 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG } 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG 10:55:32 DEBUG ] 10:55:32 DEBUG "248524800" 10:55:32 DEBUG "pod": "ds-idrepo-0" 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG }, 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 DEBUG "value": [ 10:55:32 DEBUG ] 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG } 10:55:32 DEBUG "2.6666666666666665" 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "ds-idrepo-1" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [...] 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:55:32 DEBUG 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG { 10:55:32 DEBUG "pod": "login-ui-5fdf49fc66-7c9cd" 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG }, 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "data": { 10:55:32 DEBUG "value": [ 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG "result": [ 10:55:32 DEBUG "0.00039057932500664524" 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG { 10:55:32 DEBUG ] 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG } 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "ds-cts-1" 10:55:32 DEBUG "instance": "10.142.0.24:9100" 10:55:32 DEBUG ] 10:55:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:55:32 DEBUG }, 10:55:32 DEBUG }, 10:55:32 DEBUG } 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG "value": [ 10:55:32 DEBUG } 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG [...] 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "ds-cts-0" 10:55:32 DEBUG "instance": "10.142.0.74:9100" 10:55:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:55:32 DEBUG }, 10:55:32 DEBUG }, 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG "value": [ 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0" 10:55:32 DEBUG "9614.233333333334" 10:55:32 DEBUG "0.06666666666666667" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG ] 10:55:32 DEBUG ] 10:55:32 DEBUG ] 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG ] 10:55:32 DEBUG ] 10:55:32 DEBUG ] 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "instance": "10.142.0.24:9100" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "instance": "10.142.0.74:9100" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0.00033333333333333365" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952532" 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "ds-idrepo-1" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "ds-cts-0" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:32 INFO [http_cmd]: http status code OK 10:55:32 DEBUG --- status code --- 10:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:55:32 DEBUG --- http response --- 10:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:55:32 DEBUG ----- output ----- 10:55:32 DEBUG { 10:55:32 DEBUG "status": "success", 10:55:32 DEBUG "data": { 10:55:32 DEBUG "resultType": "vector", 10:55:32 DEBUG "result": [ 10:55:32 DEBUG { 10:55:32 DEBUG "metric": { 10:55:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG [...] 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG 10:55:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:55:32 DEBUG }, 10:55:32 DEBUG "value": [ 10:55:32 DEBUG 1713952532, 10:55:32 DEBUG "0.015" 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG ] 10:55:32 DEBUG } 10:55:32 DEBUG } 10:55:44 INFO 10:55:44 INFO [loop_until]: kubectl --namespace=xlou top pods 10:55:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:55:44 INFO [loop_until]: OK (rc = 0) 10:55:44 DEBUG --- stdout --- 10:55:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 8Mi am-769b98f8d5-2pqrk 5m 3471Mi am-769b98f8d5-2rr6f 6m 3409Mi am-769b98f8d5-h2r87 3m 3436Mi ds-cts-0 66m 559Mi ds-cts-1 51m 540Mi ds-cts-2 110m 506Mi ds-idrepo-0 303m 11826Mi ds-idrepo-1 296m 11916Mi ds-idrepo-2 338m 5823Mi end-user-ui-58878d7668-zkqbp 1m 5Mi idm-6c95d7588f-98n7b 4m 4591Mi idm-6c95d7588f-j6s6c 33m 3398Mi lodemon-6b9df475cd-csxlf 2m 68Mi login-ui-5fdf49fc66-7c9cd 1m 4Mi overseer-0-55d5975bd7-gp92m 275m 329Mi 10:55:44 DEBUG --- stderr --- 10:55:44 DEBUG 10:55:49 INFO 10:55:49 INFO [loop_until]: kubectl --namespace=xlou top node 10:55:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:55:49 INFO [loop_until]: OK (rc = 0) 10:55:49 DEBUG --- stdout --- 10:55:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 397m 2% 8229Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 286m 1% 13070Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 439m 2% 13035Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 58m 0% 1741Mi 2% gke-xlou-cdm-default-pool-9d1e5395-sbmc 66m 0% 6109Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 57m 0% 1714Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 141m 0% 1622Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 87m 0% 5012Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 75m 0% 6162Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 151m 0% 2481Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 69m 0% 4850Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 62m 0% 1198Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 75m 0% 4943Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 551m 3% 3825Mi 6% 10:55:49 DEBUG --- stderr --- 10:55:49 DEBUG 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952592" 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "instance": "10.142.0.24:9100" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "instance": "10.142.0.74:9100" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "15110.966666666667" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "pod": "ds-idrepo-1" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "pod": "ds-cts-1" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "ds-cts-0" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 ERROR [http_cmd]: ERROR 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 ERROR --- status code --- 10:56:32 DEBUG --- status code --- 10:56:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 ERROR --- http response --- 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:56:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG { 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:56:32 DEBUG "data": { 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG { 10:56:32 DEBUG "result": [ 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG { 10:56:32 DEBUG "data": { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "pod": "ds-idrepo-1" 10:56:32 DEBUG "result": [ 10:56:32 DEBUG }, 10:56:32 DEBUG { 10:56:32 DEBUG "value": [ 10:56:32 DEBUG "metric": { 10:56:32 DEBUG 10:56:32 DEBUG "pod": "ds-idrepo-1" 10:56:32 DEBUG 10:56:32 DEBUG }, 10:56:32 DEBUG 10:56:32 DEBUG "value": [ 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG "pod": "ds-idrepo-0" 10:56:32 DEBUG 10:56:32 DEBUG }, 10:56:32 DEBUG 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:56:32 DEBUG "0.36666666666666664" 10:56:32 DEBUG }, 10:56:32 DEBUG ] 10:56:32 DEBUG "value": [ 10:56:32 DEBUG } 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG ] 10:56:32 DEBUG "0" 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "pod": "ds-cts-1" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "ds-cts-0" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG { 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:56:32 DEBUG "data": { 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG { 10:56:32 DEBUG "result": [ 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG { 10:56:32 DEBUG "data": { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:56:32 DEBUG "result": [ 10:56:32 DEBUG }, 10:56:32 DEBUG { 10:56:32 DEBUG "value": [ 10:56:32 DEBUG "metric": { 10:56:32 DEBUG 10:56:32 DEBUG "pod": "ds-idrepo-1" 10:56:32 DEBUG 10:56:32 DEBUG }, 10:56:32 DEBUG 10:56:32 DEBUG "value": [ 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:56:32 DEBUG 10:56:32 DEBUG }, 10:56:32 DEBUG 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:56:32 DEBUG "0" 10:56:32 DEBUG }, 10:56:32 DEBUG ] 10:56:32 DEBUG "value": [ 10:56:32 DEBUG } 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG ] 10:56:32 DEBUG "0" 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "pod": "ds-idrepo-1" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "ds-idrepo-0" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0.36666666666666664" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "instance": "10.142.0.24:9100" 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:56:32 DEBUG }, 10:56:32 DEBUG }, 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:56:32 DEBUG "value": [ 10:56:32 DEBUG "value": [ 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG { 10:56:32 DEBUG 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG 10:56:32 DEBUG "data": { 10:56:32 DEBUG [...] 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG [...] 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG 10:56:32 DEBUG "result": [ 10:56:32 DEBUG 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:56:32 DEBUG 10:56:32 DEBUG { 10:56:32 DEBUG 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "instance": "10.142.0.74:9100" 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG { 10:56:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:56:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:56:32 DEBUG }, 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG }, 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG "data": { 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:56:32 DEBUG "value": [ 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG 10:56:32 DEBUG "2893310976" 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "259309568" 10:56:32 DEBUG 10:56:32 DEBUG ] 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG ] 10:56:32 DEBUG 10:56:32 DEBUG } 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "data": { 10:56:32 DEBUG } 10:56:32 DEBUG [...] 10:56:32 DEBUG ] 10:56:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG ] 10:56:32 DEBUG 10:56:32 DEBUG } 10:56:32 DEBUG }, 10:56:32 DEBUG "result": [ 10:56:32 DEBUG } 10:56:32 DEBUG 10:56:32 DEBUG } 10:56:32 DEBUG "value": [ 10:56:32 DEBUG { 10:56:32 DEBUG } 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:56:32 DEBUG 10:56:32 DEBUG "pod": "ds-idrepo-1" 10:56:32 DEBUG }, 10:56:32 DEBUG 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG [...] 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "0" 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG ] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG } 10:56:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:56:32 DEBUG [...] 10:56:32 DEBUG ] 10:56:32 DEBUG }, 10:56:32 DEBUG 10:56:32 DEBUG } 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG } 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG 10:56:32 DEBUG "0.3592309298827258" 10:56:32 DEBUG "pod": "ds-idrepo-0" 10:56:32 DEBUG ] 10:56:32 DEBUG }, 10:56:32 DEBUG } 10:56:32 DEBUG "value": [ 10:56:32 DEBUG ] 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952592" 10:56:32 DEBUG "0.36666666666666664" 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952592" 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "instance": "10.142.0.24:9100" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "instance": "10.142.0.74:9100" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0.00006666666666662971" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "instance": "10.142.0.24:9100" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "instance": "10.142.0.74:9100" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG { 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG "data": { 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "data": { 10:56:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG }, 10:56:32 DEBUG "result": [ 10:56:32 DEBUG "value": [ 10:56:32 DEBUG { 10:56:32 DEBUG 10:56:32 DEBUG "metric": { 10:56:32 DEBUG 10:56:32 DEBUG "pod": "ds-idrepo-1" 10:56:32 DEBUG 10:56:32 DEBUG }, 10:56:32 DEBUG [...] 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:56:32 DEBUG [...] 10:56:32 DEBUG }, 10:56:32 DEBUG 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG 10:56:32 DEBUG "0" 10:56:32 DEBUG "pod": "am-769b98f8d5-2pqrk" 10:56:32 DEBUG ] 10:56:32 DEBUG }, 10:56:32 DEBUG } 10:56:32 DEBUG "value": [ 10:56:32 DEBUG ] 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG } 10:56:32 DEBUG "0" 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952592" 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG { 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG "data": { 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "data": { 10:56:32 DEBUG "pod": "ds-cts-1" 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG }, 10:56:32 DEBUG "result": [ 10:56:32 DEBUG "value": [ 10:56:32 DEBUG { 10:56:32 DEBUG 10:56:32 DEBUG "metric": { 10:56:32 DEBUG 10:56:32 DEBUG "instance": "10.142.0.24:9100" 10:56:32 DEBUG 10:56:32 DEBUG }, 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 DEBUG [...] 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "ds-cts-0" 10:56:32 DEBUG [...] 10:56:32 DEBUG }, 10:56:32 DEBUG 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG 10:56:32 DEBUG "0" 10:56:32 DEBUG "instance": "10.142.0.74:9100" 10:56:32 DEBUG ] 10:56:32 DEBUG }, 10:56:32 DEBUG } 10:56:32 DEBUG "value": [ 10:56:32 DEBUG ] 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG } 10:56:32 DEBUG "10045.666666666668" 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952592" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0.015" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:32 INFO [http_cmd]: http status code OK 10:56:32 DEBUG --- status code --- 10:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:56:32 DEBUG --- http response --- 10:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:56:32 DEBUG ----- output ----- 10:56:32 DEBUG { 10:56:32 DEBUG "status": "success", 10:56:32 DEBUG "data": { 10:56:32 DEBUG "resultType": "vector", 10:56:32 DEBUG "result": [ 10:56:32 DEBUG { 10:56:32 DEBUG "metric": { 10:56:32 DEBUG "pod": "ds-idrepo-1" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG [...] 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG 10:56:32 DEBUG "pod": "ds-cts-0" 10:56:32 DEBUG }, 10:56:32 DEBUG "value": [ 10:56:32 DEBUG 1713952592, 10:56:32 DEBUG "0" 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG ] 10:56:32 DEBUG } 10:56:32 DEBUG } 10:56:44 INFO 10:56:44 INFO [loop_until]: kubectl --namespace=xlou top pods 10:56:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:56:44 INFO [loop_until]: OK (rc = 0) 10:56:44 DEBUG --- stdout --- 10:56:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 8Mi am-769b98f8d5-2pqrk 6m 3472Mi am-769b98f8d5-2rr6f 4m 3409Mi am-769b98f8d5-h2r87 4m 3436Mi ds-cts-0 4m 559Mi ds-cts-1 5m 541Mi ds-cts-2 5m 506Mi ds-idrepo-0 75m 11788Mi ds-idrepo-1 12m 11833Mi ds-idrepo-2 11m 5825Mi end-user-ui-58878d7668-zkqbp 1m 5Mi idm-6c95d7588f-98n7b 3m 4590Mi idm-6c95d7588f-j6s6c 11m 3398Mi lodemon-6b9df475cd-csxlf 2m 68Mi login-ui-5fdf49fc66-7c9cd 1m 4Mi overseer-0-55d5975bd7-gp92m 456m 467Mi 10:56:44 DEBUG --- stderr --- 10:56:44 DEBUG 10:56:49 INFO 10:56:49 INFO [loop_until]: kubectl --namespace=xlou top node 10:56:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:56:49 INFO [loop_until]: OK (rc = 0) 10:56:49 DEBUG --- stdout --- 10:56:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 71m 0% 8245Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 52m 0% 13090Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 65m 0% 13055Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 59m 0% 1766Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 66m 0% 6114Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 56m 0% 1706Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1633Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 83m 0% 5008Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 69m 0% 6154Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 151m 0% 2477Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 64m 0% 4837Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 59m 0% 1202Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 67m 0% 4938Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 524m 3% 3720Mi 6% 10:56:49 DEBUG --- stderr --- 10:56:49 DEBUG 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952652" 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "instance": "10.142.0.24:9100" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "instance": "10.142.0.74:9100" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "14103.533333333335" 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "pod": "ds-cts-1" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "pod": "ds-cts-0" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "0" 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "pod": "ds-cts-0" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "10077.489245227607" 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952652" 10:57:32 ERROR [http_cmd]: ERROR 10:57:32 ERROR --- status code --- 10:57:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:57:32 ERROR --- http response --- 10:57:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:57:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "0" 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:57:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:57:32 DEBUG }, 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:57:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:57:32 DEBUG }, 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "0" 10:57:32 DEBUG "0" 10:57:32 DEBUG ] 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG "data": { 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG "result": [ 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "pod": "ds-idrepo-1" 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:57:32 DEBUG }, 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG "value": [ 10:57:32 DEBUG { 10:57:32 DEBUG 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG 10:57:32 DEBUG "data": { 10:57:32 DEBUG 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG [...] 10:57:32 DEBUG "result": [ 10:57:32 DEBUG 10:57:32 DEBUG { 10:57:32 DEBUG 10:57:32 DEBUG "metric": { 10:57:32 DEBUG 10:57:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:57:32 DEBUG "pod": "ds-idrepo-0" 10:57:32 DEBUG }, 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG 10:57:32 DEBUG "0" 10:57:32 DEBUG 10:57:32 DEBUG ] 10:57:32 DEBUG [...] 10:57:32 DEBUG } 10:57:32 DEBUG 10:57:32 DEBUG ] 10:57:32 DEBUG 10:57:32 DEBUG } 10:57:32 DEBUG 10:57:32 DEBUG } 10:57:32 DEBUG "pod": "ds-cts-0" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "0" 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "pod": "ds-cts-0" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "415.56656681350955" 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952652" 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952652" 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "0" 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "pod": "ds-cts-1" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "pod": "ds-cts-0" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "0.03333333333333333" 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:57:32 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:57:32 DEBUG }, 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:57:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:57:32 DEBUG }, 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "0.43115187374711134" 10:57:32 DEBUG "285126656" 10:57:32 DEBUG ] 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:57:32 DEBUG "pod": "ds-idrepo-1" 10:57:32 DEBUG }, 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:57:32 DEBUG "pod": "ds-idrepo-0" 10:57:32 DEBUG }, 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "0" 10:57:32 DEBUG "0" 10:57:32 DEBUG ] 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952652" 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952652" 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "instance": "10.142.0.24:9100" 10:57:32 DEBUG "instance": "10.142.0.24:9100" 10:57:32 DEBUG }, 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "instance": "10.142.0.74:9100" 10:57:32 DEBUG "instance": "10.142.0.74:9100" 10:57:32 DEBUG }, 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "0.0002666666666666373" 10:57:32 DEBUG "2893587456" 10:57:32 DEBUG ] 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: http status code OK 10:57:32 DEBUG --- status code --- 10:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:32 DEBUG --- http response --- 10:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:57:32 DEBUG ----- output ----- 10:57:32 DEBUG { 10:57:32 DEBUG "status": "success", 10:57:32 DEBUG "data": { 10:57:32 DEBUG "resultType": "vector", 10:57:32 DEBUG "result": [ 10:57:32 DEBUG { 10:57:32 DEBUG "metric": { 10:57:32 DEBUG "pod": "ds-idrepo-1" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG [...] 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG 10:57:32 DEBUG "pod": "ds-idrepo-0" 10:57:32 DEBUG }, 10:57:32 DEBUG "value": [ 10:57:32 DEBUG 1713952652, 10:57:32 DEBUG "0" 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG ] 10:57:32 DEBUG } 10:57:32 DEBUG } 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952652" 10:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952652" 10:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952653" 10:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952653" 10:57:33 INFO [http_cmd]: http status code OK 10:57:33 DEBUG --- status code --- 10:57:33 INFO [http_cmd]: http status code OK 10:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:33 DEBUG --- status code --- 10:57:33 DEBUG --- http response --- 10:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:33 DEBUG --- http response --- 10:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 10:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:57:33 DEBUG ----- output ----- 10:57:33 DEBUG ----- output ----- 10:57:33 DEBUG { 10:57:33 DEBUG { 10:57:33 DEBUG "status": "success", 10:57:33 DEBUG "status": "success", 10:57:33 DEBUG "data": { 10:57:33 DEBUG "data": { 10:57:33 DEBUG "resultType": "vector", 10:57:33 DEBUG "resultType": "vector", 10:57:33 DEBUG "result": [ 10:57:33 DEBUG "result": [ 10:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952653" 10:57:33 DEBUG { 10:57:33 DEBUG { 10:57:33 INFO [http_cmd]: http status code OK 10:57:33 DEBUG "metric": { 10:57:33 DEBUG "metric": { 10:57:33 DEBUG --- status code --- 10:57:33 DEBUG "pod": "admin-ui-c4c7bf584-qp6gx" 10:57:33 DEBUG "instance": "10.142.0.24:9100" 10:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:33 DEBUG }, 10:57:33 DEBUG }, 10:57:33 DEBUG --- http response --- 10:57:33 DEBUG "value": [ 10:57:33 DEBUG "value": [ 10:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG ----- output ----- 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG { 10:57:33 INFO [http_cmd]: http status code OK 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG "status": "success", 10:57:33 DEBUG --- status code --- 10:57:33 DEBUG [...] 10:57:33 DEBUG [...] 10:57:33 DEBUG "data": { 10:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG "resultType": "vector", 10:57:33 DEBUG --- http response --- 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG "result": [ 10:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG { 10:57:33 DEBUG ----- output ----- 10:57:33 DEBUG "pod": "ds-cts-0" 10:57:33 DEBUG "instance": "10.142.0.74:9100" 10:57:33 DEBUG "metric": { 10:57:33 DEBUG { 10:57:33 DEBUG }, 10:57:33 DEBUG }, 10:57:33 DEBUG "pod": "am-769b98f8d5-h2r87" 10:57:33 DEBUG "status": "success", 10:57:33 DEBUG "value": [ 10:57:33 DEBUG "value": [ 10:57:33 DEBUG }, 10:57:33 DEBUG "data": { 10:57:33 DEBUG 1713952652, 10:57:33 DEBUG 1713952652, 10:57:33 DEBUG "value": [ 10:57:33 DEBUG "resultType": "vector", 10:57:33 DEBUG "0.002245236358297049" 10:57:33 DEBUG "0" 10:57:33 DEBUG 10:57:33 DEBUG "result": [ 10:57:33 DEBUG ] 10:57:33 DEBUG ] 10:57:33 DEBUG 10:57:33 DEBUG { 10:57:33 DEBUG } 10:57:33 DEBUG } 10:57:33 DEBUG 10:57:33 DEBUG "metric": { 10:57:33 DEBUG ] 10:57:33 DEBUG ] 10:57:33 DEBUG [...] 10:57:33 DEBUG "pod": "ds-cts-1" 10:57:33 DEBUG } 10:57:33 DEBUG } 10:57:33 DEBUG 10:57:33 DEBUG }, 10:57:33 DEBUG } 10:57:33 DEBUG } 10:57:33 DEBUG 10:57:33 DEBUG "value": [ 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:57:33 DEBUG 10:57:33 DEBUG }, 10:57:33 DEBUG 10:57:33 DEBUG "value": [ 10:57:33 DEBUG [...] 10:57:33 DEBUG 1713952653, 10:57:33 DEBUG 10:57:33 DEBUG "0" 10:57:33 DEBUG 10:57:33 DEBUG ] 10:57:33 DEBUG 10:57:33 DEBUG } 10:57:33 DEBUG "pod": "ds-cts-0" 10:57:33 DEBUG ] 10:57:33 DEBUG }, 10:57:33 DEBUG } 10:57:33 DEBUG "value": [ 10:57:33 DEBUG } 10:57:33 DEBUG 1713952653, 10:57:33 DEBUG "0" 10:57:33 DEBUG ] 10:57:33 DEBUG } 10:57:33 DEBUG ] 10:57:33 INFO [http_cmd]: http status code OK 10:57:33 DEBUG } 10:57:33 DEBUG --- status code --- 10:57:33 DEBUG } 10:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:33 DEBUG --- http response --- 10:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:57:33 DEBUG ----- output ----- 10:57:33 DEBUG { 10:57:33 DEBUG "status": "success", 10:57:33 DEBUG "data": { 10:57:33 DEBUG "resultType": "vector", 10:57:33 DEBUG "result": [ 10:57:33 DEBUG { 10:57:33 DEBUG "metric": { 10:57:33 DEBUG "pod": "am-769b98f8d5-h2r87" 10:57:33 DEBUG }, 10:57:33 DEBUG "value": [ 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG [...] 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:57:33 DEBUG }, 10:57:33 DEBUG "value": [ 10:57:33 DEBUG 1713952653, 10:57:33 DEBUG "0.015" 10:57:33 DEBUG ] 10:57:33 DEBUG } 10:57:33 DEBUG ] 10:57:33 DEBUG } 10:57:33 DEBUG } 10:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952653" 10:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952653" 10:57:33 INFO [http_cmd]: http status code OK 10:57:33 DEBUG --- status code --- 10:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:33 DEBUG --- http response --- 10:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:57:33 INFO [http_cmd]: http status code OK 10:57:33 DEBUG ----- output ----- 10:57:33 DEBUG --- status code --- 10:57:33 DEBUG { 10:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:57:33 DEBUG "status": "success", 10:57:33 DEBUG --- http response --- 10:57:33 DEBUG "data": { 10:57:33 DEBUG "resultType": "vector", 10:57:33 DEBUG "result": [ 10:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:57:33 DEBUG { 10:57:33 DEBUG ----- output ----- 10:57:33 DEBUG "metric": { 10:57:33 DEBUG { 10:57:33 DEBUG "instance": "10.142.0.24:9100" 10:57:33 DEBUG "status": "success", 10:57:33 DEBUG }, 10:57:33 DEBUG "data": { 10:57:33 DEBUG "value": [ 10:57:33 DEBUG "resultType": "vector", 10:57:33 DEBUG 10:57:33 DEBUG "result": [ 10:57:33 DEBUG 10:57:33 DEBUG { 10:57:33 DEBUG 10:57:33 DEBUG "metric": { 10:57:33 DEBUG [...] 10:57:33 DEBUG "pod": "ds-idrepo-1" 10:57:33 DEBUG 10:57:33 DEBUG }, 10:57:33 DEBUG 10:57:33 DEBUG "value": [ 10:57:33 DEBUG 10:57:33 DEBUG 10:57:33 DEBUG "instance": "10.142.0.74:9100" 10:57:33 DEBUG 10:57:33 DEBUG }, 10:57:33 DEBUG 10:57:33 DEBUG "value": [ 10:57:33 DEBUG [...] 10:57:33 DEBUG 1713952653, 10:57:33 DEBUG 10:57:33 DEBUG "12486.000000000002" 10:57:33 DEBUG 10:57:33 DEBUG ] 10:57:33 DEBUG 10:57:33 DEBUG } 10:57:33 DEBUG "pod": "ds-cts-0" 10:57:33 DEBUG ] 10:57:33 DEBUG }, 10:57:33 DEBUG } 10:57:33 DEBUG "value": [ 10:57:33 DEBUG } 10:57:33 DEBUG 1713952653, 10:57:33 DEBUG "0" 10:57:33 DEBUG ] 10:57:33 DEBUG } 10:57:33 DEBUG ] 10:57:33 DEBUG } 10:57:33 DEBUG } 10:57:44 INFO 10:57:44 INFO [loop_until]: kubectl --namespace=xlou top pods 10:57:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:57:44 INFO [loop_until]: OK (rc = 0) 10:57:44 DEBUG --- stdout --- 10:57:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 8Mi am-769b98f8d5-2pqrk 4m 3472Mi am-769b98f8d5-2rr6f 4m 3409Mi am-769b98f8d5-h2r87 5m 3436Mi ds-cts-0 5m 559Mi ds-cts-1 5m 541Mi ds-cts-2 5m 506Mi ds-idrepo-0 11m 11788Mi ds-idrepo-1 11m 11833Mi ds-idrepo-2 17m 5825Mi end-user-ui-58878d7668-zkqbp 1m 5Mi idm-6c95d7588f-98n7b 3m 4590Mi idm-6c95d7588f-j6s6c 14m 3398Mi lodemon-6b9df475cd-csxlf 2m 68Mi login-ui-5fdf49fc66-7c9cd 1m 4Mi overseer-0-55d5975bd7-gp92m 774m 496Mi 10:57:44 DEBUG --- stderr --- 10:57:44 DEBUG 10:57:49 INFO 10:57:49 INFO [loop_until]: kubectl --namespace=xlou top node 10:57:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:57:50 INFO [loop_until]: OK (rc = 0) 10:57:50 DEBUG --- stdout --- 10:57:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 74m 0% 8246Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 53m 0% 13097Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 63m 0% 13063Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 59m 0% 1767Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 64m 0% 6107Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1718Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 56m 0% 1641Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 82m 0% 5006Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 70m 0% 6149Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 145m 0% 2465Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 66m 0% 4833Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 60m 0% 1196Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 76m 0% 4925Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 649m 4% 3776Mi 6% 10:57:50 DEBUG --- stderr --- 10:57:50 DEBUG 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952712" 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 DEBUG --- status code --- 10:58:32 DEBUG --- status code --- 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG { 10:58:32 DEBUG { 10:58:32 DEBUG "status": "success", 10:58:32 DEBUG "status": "success", 10:58:32 DEBUG "data": { 10:58:32 DEBUG "data": { 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG "result": [ 10:58:32 DEBUG "result": [ 10:58:32 DEBUG { 10:58:32 DEBUG { 10:58:32 DEBUG "metric": { 10:58:32 DEBUG "metric": { 10:58:32 DEBUG "pod": "ds-cts-1" 10:58:32 DEBUG "instance": "10.142.0.24:9100" 10:58:32 DEBUG }, 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG [...] 10:58:32 DEBUG [...] 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG "pod": "ds-cts-0" 10:58:32 DEBUG "instance": "10.142.0.74:9100" 10:58:32 DEBUG }, 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG "0" 10:58:32 DEBUG "12327.766666666665" 10:58:32 DEBUG ] 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 DEBUG ] 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 DEBUG --- status code --- 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG { 10:58:32 DEBUG "status": "success", 10:58:32 DEBUG "data": { 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG "result": [ 10:58:32 DEBUG { 10:58:32 DEBUG "metric": { 10:58:32 DEBUG "pod": "ds-idrepo-1" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG [...] 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG "pod": "ds-cts-2" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG "152.12343498925932" 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713952712" 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 ERROR [http_cmd]: ERROR 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 ERROR --- status code --- 10:58:32 DEBUG --- status code --- 10:58:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 ERROR --- http response --- 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:58:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG { 10:58:32 DEBUG "status": "success", 10:58:32 DEBUG "data": { 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG "result": [ 10:58:32 DEBUG { 10:58:32 DEBUG "metric": { 10:58:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG [...] 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG "0" 10:58:32 DEBUG ] 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 DEBUG } 10:58:32 DEBUG --- status code --- 10:58:32 DEBUG ] 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG } 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG } 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG { 10:58:32 DEBUG "status": "success", 10:58:32 DEBUG "data": { 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG "result": [ 10:58:32 DEBUG { 10:58:32 DEBUG "metric": { 10:58:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG [...] 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG "0" 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 DEBUG --- status code --- 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG { 10:58:32 DEBUG "status": "success", 10:58:32 DEBUG "data": { 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG "result": [ 10:58:32 DEBUG { 10:58:32 DEBUG "metric": { 10:58:32 DEBUG "pod": "ds-idrepo-1" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG [...] 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG "pod": "ds-idrepo-0" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG "0" 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 DEBUG --- status code --- 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG --- status code --- 10:58:32 DEBUG --- status code --- 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:58:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:58:32 DEBUG { 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG "status": "success", 10:58:32 DEBUG { 10:58:32 DEBUG "data": { 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 DEBUG "status": "success", 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG "data": { 10:58:32 DEBUG "result": [ 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG { 10:58:32 DEBUG "result": [ 10:58:32 DEBUG "metric": { 10:58:32 DEBUG { 10:58:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:58:32 DEBUG "metric": { 10:58:32 DEBUG }, 10:58:32 DEBUG "pod": "ds-idrepo-1" 10:58:32 DEBUG "value": [ 10:58:32 DEBUG }, 10:58:32 DEBUG 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG [...] 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG [...] 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:58:32 DEBUG 10:58:32 DEBUG }, 10:58:32 DEBUG "pod": "ds-idrepo-0" 10:58:32 DEBUG "value": [ 10:58:32 DEBUG }, 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG "0" 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG ] 10:58:32 DEBUG "0" 10:58:32 DEBUG } 10:58:32 DEBUG ] 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 DEBUG --- status code --- 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG { 10:58:32 DEBUG "status": "success", 10:58:32 DEBUG "data": { 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG "result": [ 10:58:32 DEBUG { 10:58:32 DEBUG "metric": { 10:58:32 DEBUG "pod": "ds-idrepo-1" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG [...] 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG "pod": "ds-cts-2" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG "0" 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 DEBUG --- status code --- 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG { 10:58:32 DEBUG "status": "success", 10:58:32 DEBUG "data": { 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG "result": [ 10:58:32 DEBUG { 10:58:32 DEBUG "metric": { 10:58:32 DEBUG "pod": "ds-idrepo-1" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG [...] 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG "pod": "ds-cts-2" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG "152.12343498925932" 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952712" 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713952712" 10:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713952712" 10:58:32 INFO [http_cmd]: http status code OK 10:58:32 DEBUG --- status code --- 10:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:32 DEBUG --- http response --- 10:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:58:32 DEBUG ----- output ----- 10:58:32 DEBUG { 10:58:32 DEBUG "status": "success", 10:58:32 DEBUG "data": { 10:58:32 DEBUG "resultType": "vector", 10:58:32 DEBUG "result": [ 10:58:32 DEBUG { 10:58:32 DEBUG "metric": { 10:58:32 DEBUG "pod": "am-769b98f8d5-h2r87" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG [...] 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG 10:58:32 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:58:32 DEBUG }, 10:58:32 DEBUG "value": [ 10:58:32 DEBUG 1713952712, 10:58:32 DEBUG "0" 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG ] 10:58:32 DEBUG } 10:58:32 DEBUG } 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "pod": "ds-cts-1" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "pod": "ds-cts-0" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952712, 10:58:33 DEBUG "0" 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952712, 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952713" 10:58:33 DEBUG "0.6227030775798397" 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG ] 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG } 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:33 DEBUG } 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952712, 10:58:33 DEBUG "415039488" 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "pod": "am-769b98f8d5-h2r87" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG "0" 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952713" 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952713" 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952713" 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "instance": "10.142.0.24:9100" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "instance": "10.142.0.74:9100" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG "0.0003333333333333854" 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952713" 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "instance": "10.142.0.24:9100" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "instance": "10.142.0.74:9100" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG "0" 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG } 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "pod": "am-769b98f8d5-h2r87" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG "0" 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952713" 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "instance": "10.142.0.24:9100" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "instance": "10.142.0.74:9100" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG "2893853696" 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "pod": "ds-idrepo-1" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "pod": "ds-idrepo-0" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG "0" 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952713" 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713952713" 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG --- status code --- 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 10:58:33 DEBUG { 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG { 10:58:33 DEBUG "data": { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "result": [ 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG { 10:58:33 DEBUG "result": [ 10:58:33 DEBUG "metric": { 10:58:33 DEBUG { 10:58:33 DEBUG "instance": "10.142.0.24:9100" 10:58:33 DEBUG "metric": { 10:58:33 DEBUG }, 10:58:33 DEBUG "pod": "ds-idrepo-1" 10:58:33 DEBUG "value": [ 10:58:33 DEBUG }, 10:58:33 DEBUG 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "instance": "10.142.0.74:9100" 10:58:33 DEBUG 10:58:33 DEBUG }, 10:58:33 DEBUG "pod": "ds-cts-2" 10:58:33 DEBUG "value": [ 10:58:33 DEBUG }, 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG "9341.466666666667" 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG ] 10:58:33 DEBUG "0.0028164704834148105" 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713952713" 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713952713" 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "pod": "am-769b98f8d5-h2r87" 10:58:33 DEBUG }, 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG "value": [ 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 10:58:33 DEBUG 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG 10:58:33 DEBUG { 10:58:33 DEBUG 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "pod": "am-769b98f8d5-2rr6f" 10:58:33 DEBUG "data": { 10:58:33 DEBUG }, 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "value": [ 10:58:33 DEBUG "result": [ 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG { 10:58:33 DEBUG "0.015" 10:58:33 DEBUG "metric": { 10:58:33 DEBUG ] 10:58:33 DEBUG "pod": "ds-cts-1" 10:58:33 DEBUG } 10:58:33 DEBUG }, 10:58:33 DEBUG ] 10:58:33 DEBUG "value": [ 10:58:33 DEBUG } 10:58:33 DEBUG 10:58:33 DEBUG } 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "pod": "ds-cts-0" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG "0" 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713952713" 10:58:33 INFO [http_cmd]: http status code OK 10:58:33 DEBUG --- status code --- 10:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:58:33 DEBUG --- http response --- 10:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 10:58:33 DEBUG ----- output ----- 10:58:33 DEBUG { 10:58:33 DEBUG "status": "success", 10:58:33 DEBUG "data": { 10:58:33 DEBUG "resultType": "vector", 10:58:33 DEBUG "result": [ 10:58:33 DEBUG { 10:58:33 DEBUG "metric": { 10:58:33 DEBUG "pod": "ds-idrepo-1" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG [...] 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG 10:58:33 DEBUG "pod": "ds-cts-0" 10:58:33 DEBUG }, 10:58:33 DEBUG "value": [ 10:58:33 DEBUG 1713952713, 10:58:33 DEBUG "0" 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG ] 10:58:33 DEBUG } 10:58:33 DEBUG } 10:58:44 INFO 10:58:44 INFO [loop_until]: kubectl --namespace=xlou top pods 10:58:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:58:44 INFO [loop_until]: OK (rc = 0) 10:58:44 DEBUG --- stdout --- 10:58:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-qp6gx 1m 8Mi am-769b98f8d5-2pqrk 4m 3472Mi am-769b98f8d5-2rr6f 5m 3409Mi am-769b98f8d5-h2r87 4m 3436Mi ds-cts-0 4m 560Mi ds-cts-1 5m 540Mi ds-cts-2 11m 506Mi ds-idrepo-0 12m 11788Mi ds-idrepo-1 9m 11832Mi ds-idrepo-2 10m 5826Mi end-user-ui-58878d7668-zkqbp 1m 5Mi idm-6c95d7588f-98n7b 3m 4591Mi idm-6c95d7588f-j6s6c 13m 3398Mi lodemon-6b9df475cd-csxlf 2m 68Mi login-ui-5fdf49fc66-7c9cd 1m 4Mi overseer-0-55d5975bd7-gp92m 457m 238Mi 10:58:44 DEBUG --- stderr --- 10:58:44 DEBUG 10:58:50 INFO 10:58:50 INFO [loop_until]: kubectl --namespace=xlou top node 10:58:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:58:50 INFO [loop_until]: OK (rc = 0) 10:58:50 DEBUG --- stdout --- 10:58:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 71m 0% 8254Mi 14% gke-xlou-cdm-default-pool-9d1e5395-4r9x 53m 0% 13101Mi 22% gke-xlou-cdm-default-pool-9d1e5395-jnd9 63m 0% 13073Mi 22% gke-xlou-cdm-default-pool-9d1e5395-qd5b 55m 0% 1766Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 65m 0% 6102Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 57m 0% 1724Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 61m 0% 1640Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 82m 0% 5015Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 65m 0% 6150Mi 10% gke-xlou-cdm-ds-174e6abc-k8cj 130m 0% 2459Mi 4% gke-xlou-cdm-ds-174e6abc-lzjx 63m 0% 4843Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 61m 0% 1185Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 71m 0% 4932Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 660m 4% 3727Mi 6% 10:58:50 DEBUG --- stderr --- 10:58:50 DEBUG 10:59:01 INFO Finished: True 10:59:01 INFO Waiting for threads to register finish flag 10:59:50 INFO Done. Have a nice day! :) 127.0.0.1 - - [24/Apr/2024 10:59:50] "GET /monitoring/stop HTTP/1.1" 200 - 10:59:53 WARNING File /tmp/lodemon_data-20240424_062037/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6471', ...) 127.0.0.1 - - [24/Apr/2024 10:59:58] "GET /monitoring/process HTTP/1.1" 200 -