==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-58bdc8d6d4-hh8r9 Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Sun, 14 Apr 2024 05:39:07 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=58bdc8d6d4 skaffold.dev/run-id=30957c53-e189-4f86-859f-37fdbf12893f Annotations: Status: Running IP: 10.106.46.139 IPs: IP: 10.106.46.139 Controlled By: ReplicaSet/lodemon-58bdc8d6d4 Containers: lodemon: Container ID: containerd://3643888f218b8019c478cad8ae2ab3487332c27383e8edd9f76e53da442ece48 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:06c53901427b11990b989f0a16f422c2c89c055d3393605a0d2d532652d23afc Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sun, 14 Apr 2024 05:39:08 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-scwxr (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-scwxr: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 12:44:58 INFO 12:44:58 INFO [loop_until]: kubectl --namespace=xlou top pods 12:44:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:44:58 INFO [loop_until]: OK (rc = 0) 12:44:58 DEBUG --- stdout --- 12:44:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12024m 8611Mi am-7d597cf7d6-8vr8f 11736m 8651Mi am-7d597cf7d6-9wz9h 15946m 8980Mi ds-cts-0 11556m 8120Mi ds-cts-1 11862m 8388Mi ds-cts-2 12443m 8130Mi ds-idrepo-0 13267m 4555Mi ds-idrepo-1 2083m 4318Mi ds-idrepo-2 1913m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-58bdc8d6d4-hh8r9 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6718m 1557Mi 12:44:58 DEBUG --- stderr --- 12:44:58 DEBUG 12:45:04 INFO 12:45:04 INFO [loop_until]: kubectl --namespace=xlou top node 12:45:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:45:04 INFO [loop_until]: OK (rc = 0) 12:45:04 DEBUG --- stdout --- 12:45:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 4883Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12127m 40% 10717Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12925m 43% 10679Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16997m 56% 3040Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2021m 6% 6603Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11439m 38% 11995Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2926Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12473m 41% 10103Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2026m 6% 5734Mi 5% gke-xlou-cdm-ds-0f601751-j488 11931m 39% 10333Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 72m 0% 2210Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16262m 54% 10587Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13321m 44% 6238Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6818m 42% 5150Mi 8% 12:45:04 DEBUG --- stderr --- 12:45:04 DEBUG 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095142" 12:45:42 ERROR [http_cmd]: ERROR 12:45:42 ERROR --- status code --- 12:45:42 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 ERROR --- http response --- 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:45:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG }, 12:45:42 DEBUG { 12:45:42 DEBUG "value": [ 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG 12:45:42 DEBUG "data": { 12:45:42 DEBUG 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG 12:45:42 DEBUG "result": [ 12:45:42 DEBUG [...] 12:45:42 DEBUG { 12:45:42 DEBUG 12:45:42 DEBUG "metric": { 12:45:42 DEBUG 12:45:42 DEBUG "pod": "ds-idrepo-1" 12:45:42 DEBUG 12:45:42 DEBUG }, 12:45:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:45:42 DEBUG "value": [ 12:45:42 DEBUG }, 12:45:42 DEBUG 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG 12:45:42 DEBUG "2293.0666666666666" 12:45:42 DEBUG [...] 12:45:42 DEBUG ] 12:45:42 DEBUG 12:45:42 DEBUG } 12:45:42 DEBUG 12:45:42 DEBUG ] 12:45:42 DEBUG 12:45:42 DEBUG } 12:45:42 DEBUG "pod": "ds-cts-1" 12:45:42 DEBUG } 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0.000033333333333333335" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "instance": "10.142.0.107:9100" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "instance": "10.142.0.43:9100" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "12417.233333333332" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG [...] 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:42 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG }, 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "value": [ 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "6.649195260266789" 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG ] 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG } 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG ] 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG } 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:45:42 DEBUG } 12:45:42 DEBUG "instance": "10.142.0.107:9100" 12:45:42 DEBUG }, 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:45:42 DEBUG "instance": "10.142.0.43:9100" 12:45:42 DEBUG }, 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "2313" 12:45:42 DEBUG "5351.233333333333" 12:45:42 DEBUG ] 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "ds-idrepo-1" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "ds-cts-1" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "ds-idrepo-1" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "ds-idrepo-0" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "instance": "10.142.0.107:9100" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "instance": "10.142.0.43:9100" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "13002630144" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "ds-cts-2" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "ds-idrepo-1" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "ds-idrepo-0" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "2290.4333333333334" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG { 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG "metric": { 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "ds-cts-2" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "idm-b85f9d978-rl76f" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "ds-cts-2" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "412749" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "1730056192" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0.034" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "instance": "10.142.0.107:9100" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "instance": "10.142.0.43:9100" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0.0004666666666667633" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "ds-cts-2" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "idm-b85f9d978-rl76f" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "ds-idrepo-1" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "ds-idrepo-0" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "ds-idrepo-1" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "ds-idrepo-0" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "ds-idrepo-1" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "ds-idrepo-0" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "0" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095142" 12:45:42 INFO [http_cmd]: http status code OK 12:45:42 DEBUG --- status code --- 12:45:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:42 DEBUG --- http response --- 12:45:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:45:42 DEBUG ----- output ----- 12:45:42 DEBUG { 12:45:42 DEBUG "status": "success", 12:45:42 DEBUG "data": { 12:45:42 DEBUG "resultType": "vector", 12:45:42 DEBUG "result": [ 12:45:42 DEBUG { 12:45:42 DEBUG "metric": { 12:45:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG [...] 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG 12:45:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:45:42 DEBUG }, 12:45:42 DEBUG "value": [ 12:45:42 DEBUG 1713095142, 12:45:42 DEBUG "2293.9333333333334" 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG ] 12:45:42 DEBUG } 12:45:42 DEBUG } 12:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095143" 12:45:43 INFO [http_cmd]: http status code OK 12:45:43 DEBUG --- status code --- 12:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:45:43 DEBUG --- http response --- 12:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:45:43 DEBUG ----- output ----- 12:45:43 DEBUG { 12:45:43 DEBUG "status": "success", 12:45:43 DEBUG "data": { 12:45:43 DEBUG "resultType": "vector", 12:45:43 DEBUG "result": [ 12:45:43 DEBUG { 12:45:43 DEBUG "metric": { 12:45:43 DEBUG "instance": "10.142.0.107:9100" 12:45:43 DEBUG }, 12:45:43 DEBUG "value": [ 12:45:43 DEBUG 12:45:43 DEBUG 12:45:43 DEBUG 12:45:43 DEBUG [...] 12:45:43 DEBUG 12:45:43 DEBUG 12:45:43 DEBUG 12:45:43 DEBUG "instance": "10.142.0.43:9100" 12:45:43 DEBUG }, 12:45:43 DEBUG "value": [ 12:45:43 DEBUG 1713095143, 12:45:43 DEBUG "0" 12:45:43 DEBUG ] 12:45:43 DEBUG } 12:45:43 DEBUG ] 12:45:43 DEBUG } 12:45:43 DEBUG } 12:45:58 INFO 12:45:58 INFO [loop_until]: kubectl --namespace=xlou top pods 12:45:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:45:58 INFO [loop_until]: OK (rc = 0) 12:45:58 DEBUG --- stdout --- 12:45:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12526m 8609Mi am-7d597cf7d6-8vr8f 11227m 8649Mi am-7d597cf7d6-9wz9h 15845m 8982Mi ds-cts-0 11642m 8115Mi ds-cts-1 11751m 8388Mi ds-cts-2 11694m 8128Mi ds-idrepo-0 13322m 4555Mi ds-idrepo-1 2014m 4318Mi ds-idrepo-2 1870m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6626m 1557Mi 12:45:58 DEBUG --- stderr --- 12:45:58 DEBUG 12:46:04 INFO 12:46:04 INFO [loop_until]: kubectl --namespace=xlou top node 12:46:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:46:04 INFO [loop_until]: OK (rc = 0) 12:46:04 DEBUG --- stdout --- 12:46:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4887Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11862m 39% 10721Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12907m 43% 10668Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17467m 58% 3043Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2054m 6% 6598Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11742m 39% 11983Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2931Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12243m 41% 10108Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1997m 6% 5731Mi 5% gke-xlou-cdm-ds-0f601751-j488 12225m 40% 10340Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2212Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16968m 56% 10555Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13370m 44% 6227Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6811m 42% 5153Mi 8% 12:46:04 DEBUG --- stderr --- 12:46:04 DEBUG 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "ds-idrepo-1" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "ds-cts-1" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "0.0001" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 ERROR [http_cmd]: ERROR 12:46:42 ERROR --- status code --- 12:46:42 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:46:42 ERROR --- http response --- 12:46:42 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:46:42 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "2306" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "ds-cts-2" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "0" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "2299.4" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG { 12:46:42 DEBUG { 12:46:42 DEBUG { 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG "data": { 12:46:42 DEBUG "data": { 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG "result": [ 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG { 12:46:42 DEBUG { 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "instance": "10.142.0.107:9100" 12:46:42 DEBUG "data": { 12:46:42 DEBUG "pod": "ds-idrepo-1" 12:46:42 DEBUG "instance": "10.142.0.107:9100" 12:46:42 DEBUG }, 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG }, 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG "result": [ 12:46:42 DEBUG "value": [ 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG { 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "metric": { 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG }, 12:46:42 DEBUG [...] 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "instance": "10.142.0.43:9100" 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG 12:46:42 DEBUG "pod": "ds-idrepo-0" 12:46:42 DEBUG "instance": "10.142.0.43:9100" 12:46:42 DEBUG }, 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [...] 12:46:42 DEBUG }, 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:46:42 DEBUG "value": [ 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "11844.8" 12:46:42 DEBUG 12:46:42 DEBUG { 12:46:42 DEBUG "0" 12:46:42 DEBUG "5271.833333333333" 12:46:42 DEBUG ] 12:46:42 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG ] 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG }, 12:46:42 DEBUG "data": { 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG "value": [ 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG ] 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "result": [ 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 DEBUG "6.67365460493525" 12:46:42 DEBUG { 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG "metric": { 12:46:42 DEBUG } 12:46:42 DEBUG "pod": "ds-idrepo-1" 12:46:42 DEBUG ] 12:46:42 DEBUG }, 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 DEBUG "value": [ 12:46:42 DEBUG } 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "ds-cts-1" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "0" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095202" 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "instance": "10.142.0.107:9100" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "instance": "10.142.0.43:9100" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "13003318272" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "ds-idrepo-1" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "ds-idrepo-0" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "0" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "2304.5333333333333" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "0" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG { 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "ds-cts-2" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:46:42 DEBUG 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 12:46:42 DEBUG { 12:46:42 DEBUG }, 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "value": [ 12:46:42 DEBUG "data": { 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "0.000015785456638529746" 12:46:42 DEBUG "result": [ 12:46:42 DEBUG ] 12:46:42 DEBUG { 12:46:42 DEBUG } 12:46:42 DEBUG "metric": { 12:46:42 DEBUG ] 12:46:42 DEBUG "pod": "ds-cts-2" 12:46:42 DEBUG } 12:46:42 DEBUG }, 12:46:42 DEBUG } 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "0" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "412789" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "1729748992" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "0.034" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095202" 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "instance": "10.142.0.107:9100" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "instance": "10.142.0.43:9100" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "0.0006333333333335152" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095202" 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "ds-cts-2" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "0" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "2306" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:42 INFO [http_cmd]: http status code OK 12:46:42 DEBUG --- status code --- 12:46:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:42 DEBUG --- http response --- 12:46:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:42 DEBUG ----- output ----- 12:46:42 DEBUG { 12:46:42 DEBUG "status": "success", 12:46:42 DEBUG "data": { 12:46:42 DEBUG "resultType": "vector", 12:46:42 DEBUG "result": [ 12:46:42 DEBUG { 12:46:42 DEBUG "metric": { 12:46:42 DEBUG "pod": "ds-idrepo-1" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG [...] 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG 12:46:42 DEBUG "pod": "ds-idrepo-0" 12:46:42 DEBUG }, 12:46:42 DEBUG "value": [ 12:46:42 DEBUG 1713095202, 12:46:42 DEBUG "0" 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG ] 12:46:42 DEBUG } 12:46:42 DEBUG } 12:46:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095203" 12:46:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095203" 12:46:43 INFO [http_cmd]: http status code OK 12:46:43 INFO [http_cmd]: http status code OK 12:46:43 DEBUG --- status code --- 12:46:43 DEBUG --- status code --- 12:46:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:43 DEBUG --- http response --- 12:46:43 DEBUG --- http response --- 12:46:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:43 DEBUG ----- output ----- 12:46:43 DEBUG { 12:46:43 DEBUG "status": "success", 12:46:43 DEBUG "data": { 12:46:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:46:43 DEBUG "resultType": "vector", 12:46:43 DEBUG ----- output ----- 12:46:43 DEBUG "result": [ 12:46:43 DEBUG { 12:46:43 DEBUG { 12:46:43 DEBUG "status": "success", 12:46:43 DEBUG "metric": { 12:46:43 DEBUG "data": { 12:46:43 DEBUG "pod": "ds-idrepo-1" 12:46:43 DEBUG "resultType": "vector", 12:46:43 DEBUG }, 12:46:43 DEBUG "result": [ 12:46:43 DEBUG "value": [ 12:46:43 DEBUG { 12:46:43 DEBUG 12:46:43 DEBUG "metric": { 12:46:43 DEBUG 12:46:43 DEBUG "pod": "ds-idrepo-1" 12:46:43 DEBUG 12:46:43 DEBUG }, 12:46:43 DEBUG [...] 12:46:43 DEBUG "value": [ 12:46:43 DEBUG 12:46:43 DEBUG 12:46:43 DEBUG 12:46:43 DEBUG 12:46:43 DEBUG 12:46:43 DEBUG 12:46:43 DEBUG "pod": "ds-idrepo-0" 12:46:43 DEBUG [...] 12:46:43 DEBUG }, 12:46:43 DEBUG 12:46:43 DEBUG "value": [ 12:46:43 DEBUG 12:46:43 DEBUG 1713095203, 12:46:43 DEBUG 12:46:43 DEBUG "0" 12:46:43 DEBUG "pod": "ds-idrepo-0" 12:46:43 DEBUG ] 12:46:43 DEBUG }, 12:46:43 DEBUG } 12:46:43 DEBUG "value": [ 12:46:43 DEBUG ] 12:46:43 DEBUG 1713095203, 12:46:43 DEBUG } 12:46:43 DEBUG "0" 12:46:43 DEBUG } 12:46:43 DEBUG ] 12:46:43 DEBUG } 12:46:43 DEBUG ] 12:46:43 DEBUG } 12:46:43 DEBUG } 12:46:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095203" 12:46:43 INFO [http_cmd]: http status code OK 12:46:43 DEBUG --- status code --- 12:46:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:46:43 DEBUG --- http response --- 12:46:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:46:43 DEBUG ----- output ----- 12:46:43 DEBUG { 12:46:43 DEBUG "status": "success", 12:46:43 DEBUG "data": { 12:46:43 DEBUG "resultType": "vector", 12:46:43 DEBUG "result": [ 12:46:43 DEBUG { 12:46:43 DEBUG "metric": { 12:46:43 DEBUG "instance": "10.142.0.107:9100" 12:46:43 DEBUG }, 12:46:43 DEBUG "value": [ 12:46:43 DEBUG 12:46:43 DEBUG 12:46:43 DEBUG 12:46:43 DEBUG [...] 12:46:43 DEBUG 12:46:43 DEBUG 12:46:43 DEBUG 12:46:43 DEBUG "instance": "10.142.0.43:9100" 12:46:43 DEBUG }, 12:46:43 DEBUG "value": [ 12:46:43 DEBUG 1713095203, 12:46:43 DEBUG "0.0006666666666666673" 12:46:43 DEBUG ] 12:46:43 DEBUG } 12:46:43 DEBUG ] 12:46:43 DEBUG } 12:46:43 DEBUG } 12:46:58 INFO 12:46:58 INFO [loop_until]: kubectl --namespace=xlou top pods 12:46:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:46:58 INFO [loop_until]: OK (rc = 0) 12:46:58 DEBUG --- stdout --- 12:46:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12311m 8608Mi am-7d597cf7d6-8vr8f 11506m 8650Mi am-7d597cf7d6-9wz9h 15364m 8981Mi ds-cts-0 12142m 8141Mi ds-cts-1 11936m 8388Mi ds-cts-2 11941m 8114Mi ds-idrepo-0 13372m 4555Mi ds-idrepo-1 2002m 4317Mi ds-idrepo-2 1926m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6772m 1556Mi 12:46:58 DEBUG --- stderr --- 12:46:58 DEBUG 12:47:04 INFO 12:47:04 INFO [loop_until]: kubectl --namespace=xlou top node 12:47:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:47:04 INFO [loop_until]: OK (rc = 0) 12:47:04 DEBUG --- stdout --- 12:47:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4875Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11903m 39% 10723Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12917m 43% 10663Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17356m 58% 3038Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2092m 7% 6599Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11714m 39% 11986Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2928Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12397m 41% 10115Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1973m 6% 5734Mi 5% gke-xlou-cdm-ds-0f601751-j488 12143m 40% 10344Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2224Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16597m 55% 10567Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13229m 44% 6225Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6754m 42% 5149Mi 8% 12:47:04 DEBUG --- stderr --- 12:47:04 DEBUG 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "data": { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "result": [ 12:47:42 DEBUG { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "pod": "ds-idrepo-1" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG [...] 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG "pod": "ds-cts-1" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG "0.00006666666666666667" 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 ERROR [http_cmd]: ERROR 12:47:42 ERROR --- status code --- 12:47:42 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:47:42 ERROR --- http response --- 12:47:42 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:47:42 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "data": { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "result": [ 12:47:42 DEBUG { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG [...] 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG "2304.2" 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "data": { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:47:42 DEBUG "result": [ 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG { 12:47:42 DEBUG { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "pod": "ds-cts-2" 12:47:42 DEBUG "data": { 12:47:42 DEBUG }, 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "value": [ 12:47:42 DEBUG "result": [ 12:47:42 DEBUG 12:47:42 DEBUG { 12:47:42 DEBUG 12:47:42 DEBUG "metric": { 12:47:42 DEBUG 12:47:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:47:42 DEBUG [...] 12:47:42 DEBUG }, 12:47:42 DEBUG 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 12:47:42 DEBUG 12:47:42 DEBUG }, 12:47:42 DEBUG [...] 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG 12:47:42 DEBUG "0" 12:47:42 DEBUG 12:47:42 DEBUG ] 12:47:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:47:42 DEBUG } 12:47:42 DEBUG }, 12:47:42 DEBUG ] 12:47:42 DEBUG "value": [ 12:47:42 DEBUG } 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG } 12:47:42 DEBUG "2299.6666666666665" 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "data": { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "result": [ 12:47:42 DEBUG { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "pod": "ds-idrepo-1" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG [...] 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG "pod": "ds-idrepo-0" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG "0" 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG { 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG "data": { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "result": [ 12:47:42 DEBUG { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:47:42 DEBUG }, 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:47:42 DEBUG "value": [ 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG 12:47:42 DEBUG { 12:47:42 DEBUG 12:47:42 DEBUG "status": "success", 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG 12:47:42 DEBUG "data": { 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG [...] 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095262" 12:47:42 DEBUG "result": [ 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG { 12:47:42 DEBUG 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG "instance": "10.142.0.107:9100" 12:47:42 DEBUG }, 12:47:42 DEBUG { 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG "data": { 12:47:42 DEBUG { 12:47:42 DEBUG 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG "6.628443517045901" 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG ] 12:47:42 DEBUG "result": [ 12:47:42 DEBUG "data": { 12:47:42 DEBUG 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG } 12:47:42 DEBUG { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG [...] 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG ] 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "result": [ 12:47:42 DEBUG 12:47:42 DEBUG } 12:47:42 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:47:42 DEBUG "pod": "ds-idrepo-1" 12:47:42 DEBUG { 12:47:42 DEBUG 12:47:42 DEBUG } 12:47:42 DEBUG }, 12:47:42 DEBUG "metric": { 12:47:42 DEBUG 12:47:42 DEBUG "value": [ 12:47:42 DEBUG "instance": "10.142.0.107:9100" 12:47:42 DEBUG "instance": "10.142.0.43:9100" 12:47:42 DEBUG 12:47:42 DEBUG }, 12:47:42 DEBUG }, 12:47:42 DEBUG 12:47:42 DEBUG "value": [ 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG [...] 12:47:42 DEBUG 12:47:42 DEBUG "5357.066666666667" 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG ] 12:47:42 DEBUG 12:47:42 DEBUG [...] 12:47:42 DEBUG } 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG ] 12:47:42 DEBUG "pod": "ds-cts-1" 12:47:42 DEBUG 12:47:42 DEBUG } 12:47:42 DEBUG }, 12:47:42 DEBUG 12:47:42 DEBUG } 12:47:42 DEBUG "value": [ 12:47:42 DEBUG "instance": "10.142.0.43:9100" 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG }, 12:47:42 DEBUG "0" 12:47:42 DEBUG "value": [ 12:47:42 DEBUG ] 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG } 12:47:42 DEBUG "11935.433333333332" 12:47:42 DEBUG ] 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "data": { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "result": [ 12:47:42 DEBUG { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "instance": "10.142.0.107:9100" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG [...] 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG "instance": "10.142.0.43:9100" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG "13003949056" 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "data": { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "result": [ 12:47:42 DEBUG { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "pod": "ds-idrepo-1" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG [...] 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG "pod": "ds-idrepo-0" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG "0" 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "data": { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "result": [ 12:47:42 DEBUG { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG [...] 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG "2296.9" 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095262" 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "data": { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "result": [ 12:47:42 DEBUG { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG [...] 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:47:42 DEBUG 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG { 12:47:42 DEBUG 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG "data": { 12:47:42 DEBUG }, 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:47:42 DEBUG "value": [ 12:47:42 DEBUG "result": [ 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG { 12:47:42 DEBUG { 12:47:42 DEBUG "0" 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG ] 12:47:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:47:42 DEBUG "data": { 12:47:42 DEBUG } 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG }, 12:47:42 DEBUG "resultType": "vector", 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG ] 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG "value": [ 12:47:42 DEBUG "result": [ 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG } 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG 12:47:42 DEBUG { 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG } 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG 12:47:42 DEBUG "metric": { 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 12:47:42 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:47:42 INFO [http_cmd]: http status code OK 12:47:42 DEBUG [...] 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG }, 12:47:42 DEBUG --- status code --- 12:47:42 DEBUG 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG { 12:47:42 DEBUG "value": [ 12:47:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:42 DEBUG 12:47:42 DEBUG { 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG 12:47:42 DEBUG --- http response --- 12:47:42 DEBUG 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG "data": { 12:47:42 DEBUG 12:47:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:47:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:47:42 DEBUG "data": { 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG 12:47:42 DEBUG }, 12:47:42 DEBUG ----- output ----- 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "result": [ 12:47:42 DEBUG [...] 12:47:42 DEBUG "value": [ 12:47:42 DEBUG { 12:47:42 DEBUG "result": [ 12:47:42 DEBUG { 12:47:42 DEBUG 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG "status": "success", 12:47:42 DEBUG { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG 12:47:42 DEBUG "412038.5" 12:47:42 DEBUG "data": { 12:47:42 DEBUG "metric": { 12:47:42 DEBUG "pod": "ds-cts-2" 12:47:42 DEBUG 12:47:42 DEBUG ] 12:47:42 DEBUG "resultType": "vector", 12:47:42 DEBUG "pod": "ds-cts-2" 12:47:42 DEBUG }, 12:47:42 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:47:42 DEBUG } 12:47:42 DEBUG "result": [ 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG }, 12:47:42 DEBUG ] 12:47:42 DEBUG { 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG "value": [ 12:47:42 DEBUG } 12:47:42 DEBUG "metric": { 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG } 12:47:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG "1728778240" 12:47:42 DEBUG }, 12:47:42 DEBUG 12:47:42 DEBUG [...] 12:47:42 DEBUG ] 12:47:42 DEBUG "value": [ 12:47:42 DEBUG [...] 12:47:42 DEBUG 12:47:42 DEBUG } 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG ] 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG } 12:47:42 DEBUG 12:47:42 DEBUG 12:47:42 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 12:47:42 DEBUG } 12:47:42 DEBUG [...] 12:47:42 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 12:47:42 DEBUG }, 12:47:42 DEBUG 12:47:42 DEBUG }, 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 12:47:42 DEBUG "value": [ 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG "0" 12:47:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:47:42 DEBUG "13.287718052370158" 12:47:42 DEBUG ] 12:47:42 DEBUG }, 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG "value": [ 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG 1713095262, 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG "0.034" 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 DEBUG ] 12:47:42 DEBUG } 12:47:42 DEBUG } 12:47:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095262" 12:47:43 INFO [http_cmd]: http status code OK 12:47:43 DEBUG --- status code --- 12:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:43 DEBUG --- http response --- 12:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:47:43 DEBUG ----- output ----- 12:47:43 DEBUG { 12:47:43 DEBUG "status": "success", 12:47:43 DEBUG "data": { 12:47:43 DEBUG "resultType": "vector", 12:47:43 DEBUG "result": [ 12:47:43 DEBUG { 12:47:43 DEBUG "metric": { 12:47:43 DEBUG "instance": "10.142.0.107:9100" 12:47:43 DEBUG }, 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG [...] 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG "instance": "10.142.0.43:9100" 12:47:43 DEBUG }, 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 1713095262, 12:47:43 DEBUG "0.0005333333333335114" 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG } 12:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095263" 12:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095263" 12:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095263" 12:47:43 INFO [http_cmd]: http status code OK 12:47:43 DEBUG --- status code --- 12:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:43 DEBUG --- http response --- 12:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:47:43 DEBUG ----- output ----- 12:47:43 DEBUG { 12:47:43 DEBUG "status": "success", 12:47:43 DEBUG "data": { 12:47:43 DEBUG "resultType": "vector", 12:47:43 DEBUG "result": [ 12:47:43 DEBUG { 12:47:43 DEBUG "metric": { 12:47:43 DEBUG "pod": "ds-idrepo-1" 12:47:43 DEBUG }, 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG [...] 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG "pod": "ds-idrepo-0" 12:47:43 DEBUG }, 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 1713095263, 12:47:43 DEBUG "0" 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG } 12:47:43 INFO [http_cmd]: http status code OK 12:47:43 DEBUG --- status code --- 12:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:43 DEBUG --- http response --- 12:47:43 INFO [http_cmd]: http status code OK 12:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 12:47:43 DEBUG --- status code --- 12:47:43 DEBUG ----- output ----- 12:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:43 DEBUG { 12:47:43 DEBUG --- http response --- 12:47:43 DEBUG "status": "success", 12:47:43 DEBUG "data": { 12:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:47:43 DEBUG "resultType": "vector", 12:47:43 DEBUG ----- output ----- 12:47:43 DEBUG "result": [ 12:47:43 DEBUG { 12:47:43 DEBUG { 12:47:43 DEBUG "status": "success", 12:47:43 DEBUG "metric": { 12:47:43 DEBUG "data": { 12:47:43 DEBUG "pod": "ds-cts-2" 12:47:43 DEBUG "resultType": "vector", 12:47:43 DEBUG }, 12:47:43 DEBUG "result": [ 12:47:43 DEBUG "value": [ 12:47:43 DEBUG { 12:47:43 DEBUG 12:47:43 DEBUG "metric": { 12:47:43 DEBUG 12:47:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:47:43 DEBUG 12:47:43 DEBUG }, 12:47:43 DEBUG [...] 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 12:47:43 DEBUG [...] 12:47:43 DEBUG }, 12:47:43 DEBUG 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 12:47:43 DEBUG 1713095263, 12:47:43 DEBUG 12:47:43 DEBUG "38121970.28454371" 12:47:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:47:43 DEBUG ] 12:47:43 DEBUG }, 12:47:43 DEBUG } 12:47:43 DEBUG "value": [ 12:47:43 DEBUG ] 12:47:43 DEBUG 1713095263, 12:47:43 DEBUG } 12:47:43 DEBUG "2304.3" 12:47:43 DEBUG } 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG } 12:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095263" 12:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095263" 12:47:43 INFO [http_cmd]: http status code OK 12:47:43 DEBUG --- status code --- 12:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:43 DEBUG --- http response --- 12:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:47:43 DEBUG ----- output ----- 12:47:43 DEBUG { 12:47:43 DEBUG "status": "success", 12:47:43 DEBUG "data": { 12:47:43 DEBUG "resultType": "vector", 12:47:43 DEBUG "result": [ 12:47:43 DEBUG { 12:47:43 DEBUG "metric": { 12:47:43 DEBUG "pod": "ds-idrepo-1" 12:47:43 DEBUG }, 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG [...] 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG "pod": "ds-idrepo-0" 12:47:43 DEBUG }, 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 1713095263, 12:47:43 DEBUG "0" 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG } 12:47:43 INFO [http_cmd]: http status code OK 12:47:43 DEBUG --- status code --- 12:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:43 DEBUG --- http response --- 12:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:47:43 DEBUG ----- output ----- 12:47:43 DEBUG { 12:47:43 DEBUG "status": "success", 12:47:43 DEBUG "data": { 12:47:43 DEBUG "resultType": "vector", 12:47:43 DEBUG "result": [ 12:47:43 DEBUG { 12:47:43 DEBUG "metric": { 12:47:43 DEBUG "pod": "ds-idrepo-1" 12:47:43 DEBUG }, 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG [...] 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG "pod": "ds-idrepo-0" 12:47:43 DEBUG }, 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 1713095263, 12:47:43 DEBUG "0" 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG } 12:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095263" 12:47:43 INFO [http_cmd]: http status code OK 12:47:43 DEBUG --- status code --- 12:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:47:43 DEBUG --- http response --- 12:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:47:43 DEBUG ----- output ----- 12:47:43 DEBUG { 12:47:43 DEBUG "status": "success", 12:47:43 DEBUG "data": { 12:47:43 DEBUG "resultType": "vector", 12:47:43 DEBUG "result": [ 12:47:43 DEBUG { 12:47:43 DEBUG "metric": { 12:47:43 DEBUG "instance": "10.142.0.107:9100" 12:47:43 DEBUG }, 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG [...] 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG 12:47:43 DEBUG "instance": "10.142.0.43:9100" 12:47:43 DEBUG }, 12:47:43 DEBUG "value": [ 12:47:43 DEBUG 1713095263, 12:47:43 DEBUG "0" 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG ] 12:47:43 DEBUG } 12:47:43 DEBUG } 12:47:58 INFO 12:47:58 INFO [loop_until]: kubectl --namespace=xlou top pods 12:47:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:47:58 INFO [loop_until]: OK (rc = 0) 12:47:58 DEBUG --- stdout --- 12:47:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12566m 8608Mi am-7d597cf7d6-8vr8f 11566m 8652Mi am-7d597cf7d6-9wz9h 15733m 8981Mi ds-cts-0 11825m 8116Mi ds-cts-1 11839m 8389Mi ds-cts-2 12267m 8140Mi ds-idrepo-0 13258m 4555Mi ds-idrepo-1 1995m 4318Mi ds-idrepo-2 1852m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 5m 1012Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6707m 1556Mi 12:47:58 DEBUG --- stderr --- 12:47:58 DEBUG 12:48:04 INFO 12:48:04 INFO [loop_until]: kubectl --namespace=xlou top node 12:48:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:48:04 INFO [loop_until]: OK (rc = 0) 12:48:04 DEBUG --- stdout --- 12:48:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4873Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12308m 41% 10737Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13047m 43% 10663Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17580m 58% 3050Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2142m 7% 6584Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11723m 39% 11993Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2938Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12569m 42% 10099Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1989m 6% 5747Mi 5% gke-xlou-cdm-ds-0f601751-j488 12203m 40% 10346Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2221Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16709m 55% 10568Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13410m 44% 6238Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6826m 42% 5151Mi 8% 12:48:04 DEBUG --- stderr --- 12:48:04 DEBUG 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095322" 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095322" 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095322" 12:48:42 ERROR [http_cmd]: ERROR 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 ERROR --- status code --- 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 DEBUG --- status code --- 12:48:42 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 ERROR --- http response --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:48:42 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG { 12:48:42 DEBUG { 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "data": { 12:48:42 DEBUG "data": { 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "result": [ 12:48:42 DEBUG "result": [ 12:48:42 DEBUG { 12:48:42 DEBUG { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:48:42 DEBUG "pod": "ds-idrepo-1" 12:48:42 DEBUG }, 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG [...] 12:48:42 DEBUG [...] 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:48:42 DEBUG "pod": "ds-cts-1" 12:48:42 DEBUG }, 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG "2286.5333333333333" 12:48:42 DEBUG "0.00006666666666666667" 12:48:42 DEBUG ] 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 DEBUG ] 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095322" 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095322" 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095322" 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095322" 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG { 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "data": { 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "result": [ 12:48:42 DEBUG { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "pod": "ds-idrepo-1" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG [...] 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG "pod": "ds-idrepo-0" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG "0" 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG { 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "data": { 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "result": [ 12:48:42 DEBUG { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "instance": "10.142.0.107:9100" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG [...] 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG "instance": "10.142.0.43:9100" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG "5539.8" 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG { 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "data": { 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "result": [ 12:48:42 DEBUG { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG [...] 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG "6.690395971877578" 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG { 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "data": { 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "result": [ 12:48:42 DEBUG { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "pod": "ds-idrepo-1" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG [...] 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG "pod": "ds-cts-1" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG "0" 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095322" 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095322" 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095322" 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095322" 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- http response --- 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG { 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "data": { 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "result": [ 12:48:42 DEBUG { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "pod": "ds-cts-2" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG [...] 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG "pod": "ds-idrepo-2" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG "4906.3965228105" 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095322" 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG { 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "data": { 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "result": [ 12:48:42 DEBUG { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG [...] 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG "2290.1" 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG { 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "data": { 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "result": [ 12:48:42 DEBUG { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "instance": "10.142.0.107:9100" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG [...] 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG "instance": "10.142.0.43:9100" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG "12263.93333333333" 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG { 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "data": { 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "result": [ 12:48:42 DEBUG { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "instance": "10.142.0.107:9100" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG [...] 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG "instance": "10.142.0.43:9100" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG "13004618752" 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095322" 12:48:42 INFO [http_cmd]: http status code OK 12:48:42 DEBUG --- status code --- 12:48:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:42 DEBUG --- http response --- 12:48:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:48:42 DEBUG ----- output ----- 12:48:42 DEBUG { 12:48:42 DEBUG "status": "success", 12:48:42 DEBUG "data": { 12:48:42 DEBUG "resultType": "vector", 12:48:42 DEBUG "result": [ 12:48:42 DEBUG { 12:48:42 DEBUG "metric": { 12:48:42 DEBUG "pod": "ds-idrepo-1" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG [...] 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG 12:48:42 DEBUG "pod": "ds-idrepo-0" 12:48:42 DEBUG }, 12:48:42 DEBUG "value": [ 12:48:42 DEBUG 1713095322, 12:48:42 DEBUG "0" 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG ] 12:48:42 DEBUG } 12:48:42 DEBUG } 12:48:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095322" 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095323" 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG { 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "data": { 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG "result": [ 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG [...] 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 1713095322, 12:48:43 DEBUG "2287" 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG { 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "data": { 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG "result": [ 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG [...] 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG "0" 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095323" 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095323" 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095323" 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095323" 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095323" 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095323" 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG --- http response --- 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG { 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "data": { 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG "result": [ 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "instance": "10.142.0.107:9100" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG [...] 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "instance": "10.142.0.43:9100" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG "0.00036666666666675953" 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG { 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG "data": { 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG "result": [ 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG [...] 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG "412825" 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG { 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "data": { 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG "result": [ 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG [...] 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG "0.035500000000000004" 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG --- http response --- 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG { 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG "data": { 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG "result": [ 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG { 12:48:43 DEBUG { 12:48:43 DEBUG { 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "data": { 12:48:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:48:43 DEBUG "data": { 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG }, 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG "result": [ 12:48:43 DEBUG "value": [ 12:48:43 DEBUG "result": [ 12:48:43 DEBUG { 12:48:43 DEBUG 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "pod": "ds-cts-2" 12:48:43 DEBUG 12:48:43 DEBUG "pod": "ds-cts-2" 12:48:43 DEBUG }, 12:48:43 DEBUG [...] 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:48:43 DEBUG 12:48:43 DEBUG [...] 12:48:43 DEBUG }, 12:48:43 DEBUG [...] 12:48:43 DEBUG 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "1728528384" 12:48:43 DEBUG 12:48:43 DEBUG "pod": "ds-idrepo-2" 12:48:43 DEBUG ] 12:48:43 DEBUG "pod": "ds-idrepo-2" 12:48:43 DEBUG }, 12:48:43 DEBUG } 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG ] 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG } 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG "1.8943006605290678" 12:48:43 DEBUG } 12:48:43 DEBUG "0" 12:48:43 DEBUG ] 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095323" 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095323" 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095323" 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095323" 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG { 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "data": { 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:48:43 DEBUG "result": [ 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG { 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "pod": "ds-idrepo-1" 12:48:43 DEBUG "data": { 12:48:43 DEBUG }, 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG "value": [ 12:48:43 DEBUG "result": [ 12:48:43 DEBUG 12:48:43 DEBUG { 12:48:43 DEBUG 12:48:43 DEBUG "metric": { 12:48:43 DEBUG 12:48:43 DEBUG "pod": "ds-idrepo-1" 12:48:43 DEBUG [...] 12:48:43 DEBUG }, 12:48:43 DEBUG 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "pod": "ds-idrepo-0" 12:48:43 DEBUG 12:48:43 DEBUG }, 12:48:43 DEBUG [...] 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG 12:48:43 DEBUG "0" 12:48:43 DEBUG 12:48:43 DEBUG ] 12:48:43 DEBUG "pod": "ds-idrepo-0" 12:48:43 DEBUG } 12:48:43 DEBUG }, 12:48:43 DEBUG ] 12:48:43 DEBUG "value": [ 12:48:43 DEBUG } 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG } 12:48:43 DEBUG "0" 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG { 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG "data": { 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG "result": [ 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG [...] 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG "2286.5666666666666" 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG { 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "data": { 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG "result": [ 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG [...] 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "pod": "ds-idrepo-2" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG "147469036.9964749" 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095323" 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG { 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "data": { 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG "result": [ 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "pod": "ds-idrepo-1" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG [...] 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "pod": "ds-idrepo-0" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG "0" 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095323" 12:48:43 INFO [http_cmd]: http status code OK 12:48:43 DEBUG --- status code --- 12:48:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:48:43 DEBUG --- http response --- 12:48:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:48:43 DEBUG ----- output ----- 12:48:43 DEBUG { 12:48:43 DEBUG "status": "success", 12:48:43 DEBUG "data": { 12:48:43 DEBUG "resultType": "vector", 12:48:43 DEBUG "result": [ 12:48:43 DEBUG { 12:48:43 DEBUG "metric": { 12:48:43 DEBUG "instance": "10.142.0.107:9100" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG [...] 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG 12:48:43 DEBUG "instance": "10.142.0.43:9100" 12:48:43 DEBUG }, 12:48:43 DEBUG "value": [ 12:48:43 DEBUG 1713095323, 12:48:43 DEBUG "0.00033333333333333365" 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG ] 12:48:43 DEBUG } 12:48:43 DEBUG } 12:48:58 INFO 12:48:58 INFO [loop_until]: kubectl --namespace=xlou top pods 12:48:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:48:58 INFO [loop_until]: OK (rc = 0) 12:48:58 DEBUG --- stdout --- 12:48:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12383m 8670Mi am-7d597cf7d6-8vr8f 11354m 8652Mi am-7d597cf7d6-9wz9h 15377m 8983Mi ds-cts-0 11942m 8141Mi ds-cts-1 12111m 8395Mi ds-cts-2 11900m 8107Mi ds-idrepo-0 12940m 4555Mi ds-idrepo-1 1987m 4318Mi ds-idrepo-2 1861m 4106Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6650m 1557Mi 12:48:58 DEBUG --- stderr --- 12:48:58 DEBUG 12:49:04 INFO 12:49:04 INFO [loop_until]: kubectl --namespace=xlou top node 12:49:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:49:05 INFO [loop_until]: OK (rc = 0) 12:49:05 DEBUG --- stdout --- 12:49:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 4866Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12023m 40% 10725Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13019m 43% 10729Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17469m 58% 3047Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2107m 7% 6603Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11867m 39% 12009Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2931Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12468m 41% 10105Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1998m 6% 5733Mi 5% gke-xlou-cdm-ds-0f601751-j488 12024m 40% 10345Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2227Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16835m 56% 10567Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13343m 44% 6235Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6885m 43% 5151Mi 8% 12:49:05 DEBUG --- stderr --- 12:49:05 DEBUG 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095382" 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095382" 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095382" 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095382" 12:49:42 ERROR [http_cmd]: ERROR 12:49:42 ERROR --- status code --- 12:49:42 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:49:42 ERROR --- http response --- 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG --- status code --- 12:49:42 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG { 12:49:42 DEBUG { 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG "data": { 12:49:42 DEBUG "data": { 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG "result": [ 12:49:42 DEBUG "result": [ 12:49:42 DEBUG { 12:49:42 DEBUG { 12:49:42 DEBUG "metric": { 12:49:42 DEBUG "metric": { 12:49:42 DEBUG "pod": "ds-idrepo-1" 12:49:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:49:42 DEBUG }, 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG [...] 12:49:42 DEBUG [...] 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG "pod": "ds-cts-1" 12:49:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:49:42 DEBUG }, 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG "0.00006666666666666667" 12:49:42 DEBUG "2276.866666666667" 12:49:42 DEBUG ] 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 DEBUG ] 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG { 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG "data": { 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG "result": [ 12:49:42 DEBUG { 12:49:42 DEBUG "metric": { 12:49:42 DEBUG "instance": "10.142.0.107:9100" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG [...] 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG "instance": "10.142.0.43:9100" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG "7061.4" 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095382" 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095382" 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095382" 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095382" 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG { 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG "data": { 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG "result": [ 12:49:42 DEBUG { 12:49:42 DEBUG "metric": { 12:49:42 DEBUG "pod": "ds-idrepo-1" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG [...] 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG "pod": "ds-cts-1" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG "0" 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG { 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG "data": { 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG "result": [ 12:49:42 DEBUG { 12:49:42 DEBUG "metric": { 12:49:42 DEBUG "pod": "ds-cts-2" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG [...] 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG "pod": "lodemon-58bdc8d6d4-hh8r9" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG "0" 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG { 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG "data": { 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG "result": [ 12:49:42 DEBUG { 12:49:42 DEBUG "metric": { 12:49:42 DEBUG "pod": "ds-idrepo-1" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG [...] 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG "pod": "ds-idrepo-0" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG "0" 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG { 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG "data": { 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG "result": [ 12:49:42 DEBUG { 12:49:42 DEBUG "metric": { 12:49:42 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG [...] 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG "6.753635769145345" 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095382" 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095382" 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095382" 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095382" 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG { 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG "data": { 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG "result": [ 12:49:42 DEBUG { 12:49:42 DEBUG "metric": { 12:49:42 DEBUG "instance": "10.142.0.107:9100" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG [...] 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG "instance": "10.142.0.43:9100" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG "12395.199999999999" 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG --- http response --- 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG { 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG "data": { 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:42 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG "result": [ 12:49:42 DEBUG { 12:49:42 DEBUG { 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG "metric": { 12:49:42 DEBUG "data": { 12:49:42 DEBUG "instance": "10.142.0.107:9100" 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG }, 12:49:42 DEBUG "result": [ 12:49:42 DEBUG "value": [ 12:49:42 DEBUG { 12:49:42 DEBUG 12:49:42 DEBUG "metric": { 12:49:42 DEBUG 12:49:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:49:42 DEBUG 12:49:42 DEBUG }, 12:49:42 DEBUG [...] 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG "instance": "10.142.0.43:9100" 12:49:42 DEBUG [...] 12:49:42 DEBUG }, 12:49:42 DEBUG 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG 12:49:42 DEBUG "13005136896" 12:49:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:49:42 DEBUG ] 12:49:42 DEBUG }, 12:49:42 DEBUG } 12:49:42 DEBUG "value": [ 12:49:42 DEBUG ] 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG } 12:49:42 DEBUG "2304.7666666666664" 12:49:42 DEBUG } 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095382" 12:49:42 INFO [http_cmd]: http status code OK 12:49:42 DEBUG --- status code --- 12:49:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:42 DEBUG --- http response --- 12:49:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:49:42 DEBUG ----- output ----- 12:49:42 DEBUG { 12:49:42 DEBUG "status": "success", 12:49:42 DEBUG "data": { 12:49:42 DEBUG "resultType": "vector", 12:49:42 DEBUG "result": [ 12:49:42 DEBUG { 12:49:42 DEBUG "metric": { 12:49:42 DEBUG "pod": "ds-idrepo-1" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG [...] 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG 12:49:42 DEBUG "pod": "ds-idrepo-0" 12:49:42 DEBUG }, 12:49:42 DEBUG "value": [ 12:49:42 DEBUG 1713095382, 12:49:42 DEBUG "0" 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG ] 12:49:42 DEBUG } 12:49:42 DEBUG } 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG { 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "data": { 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "result": [ 12:49:43 DEBUG { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG [...] 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "pod": "ds-idrepo-0" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "13.283221528662967" 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG { 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "data": { 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "result": [ 12:49:43 DEBUG { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG [...] 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "2281.6666666666665" 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG { 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "data": { 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "result": [ 12:49:43 DEBUG { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG [...] 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "0" 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095383" 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095383" 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG { 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "data": { 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "result": [ 12:49:43 DEBUG { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "instance": "10.142.0.107:9100" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG [...] 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "instance": "10.142.0.43:9100" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "0.00040000000000001514" 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG { 12:49:43 DEBUG { 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "data": { 12:49:43 DEBUG "data": { 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "result": [ 12:49:43 DEBUG "result": [ 12:49:43 DEBUG { 12:49:43 DEBUG { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:49:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:49:43 DEBUG }, 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG [...] 12:49:43 DEBUG [...] 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:49:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:49:43 DEBUG }, 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "0.036" 12:49:43 DEBUG "411931" 12:49:43 DEBUG ] 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 DEBUG ] 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG { 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "data": { 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "result": [ 12:49:43 DEBUG { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG [...] 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "pod": "ds-idrepo-0" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "0" 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG { 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "data": { 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "result": [ 12:49:43 DEBUG { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG [...] 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "1728290816" 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG { 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG "data": { 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG "result": [ 12:49:43 DEBUG { 12:49:43 DEBUG { 12:49:43 DEBUG { 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "data": { 12:49:43 DEBUG "data": { 12:49:43 DEBUG "pod": "ds-idrepo-1" 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG }, 12:49:43 DEBUG "result": [ 12:49:43 DEBUG "result": [ 12:49:43 DEBUG "value": [ 12:49:43 DEBUG { 12:49:43 DEBUG { 12:49:43 DEBUG 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG 12:49:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:49:43 DEBUG "pod": "ds-idrepo-1" 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG 12:49:43 DEBUG }, 12:49:43 DEBUG }, 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG [...] 12:49:43 DEBUG "value": [ 12:49:43 DEBUG "value": [ 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "pod": "ds-idrepo-0" 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG [...] 12:49:43 DEBUG [...] 12:49:43 DEBUG }, 12:49:43 DEBUG { 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "value": [ 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "data": { 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "0" 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:49:43 DEBUG "pod": "ds-idrepo-0" 12:49:43 DEBUG ] 12:49:43 DEBUG "result": [ 12:49:43 DEBUG }, 12:49:43 DEBUG }, 12:49:43 DEBUG } 12:49:43 DEBUG { 12:49:43 DEBUG "value": [ 12:49:43 DEBUG "value": [ 12:49:43 DEBUG ] 12:49:43 DEBUG "metric": { 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG } 12:49:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:49:43 DEBUG "2276.0333333333333" 12:49:43 DEBUG "0" 12:49:43 DEBUG } 12:49:43 DEBUG }, 12:49:43 DEBUG ] 12:49:43 DEBUG ] 12:49:43 DEBUG "value": [ 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 DEBUG 12:49:43 DEBUG ] 12:49:43 DEBUG ] 12:49:43 DEBUG 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 DEBUG 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 DEBUG [...] 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "pod": "ds-idrepo-0" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "152191706.4968153" 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095383" 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG { 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "data": { 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "result": [ 12:49:43 DEBUG { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "pod": "ds-idrepo-1" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG [...] 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "pod": "ds-idrepo-0" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "0" 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095383" 12:49:43 INFO [http_cmd]: http status code OK 12:49:43 DEBUG --- status code --- 12:49:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:49:43 DEBUG --- http response --- 12:49:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:49:43 DEBUG ----- output ----- 12:49:43 DEBUG { 12:49:43 DEBUG "status": "success", 12:49:43 DEBUG "data": { 12:49:43 DEBUG "resultType": "vector", 12:49:43 DEBUG "result": [ 12:49:43 DEBUG { 12:49:43 DEBUG "metric": { 12:49:43 DEBUG "instance": "10.142.0.107:9100" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG [...] 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG 12:49:43 DEBUG "instance": "10.142.0.43:9100" 12:49:43 DEBUG }, 12:49:43 DEBUG "value": [ 12:49:43 DEBUG 1713095383, 12:49:43 DEBUG "0.0003333333333333299" 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG ] 12:49:43 DEBUG } 12:49:43 DEBUG } 12:49:58 INFO 12:49:58 INFO [loop_until]: kubectl --namespace=xlou top pods 12:49:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:49:58 INFO [loop_until]: OK (rc = 0) 12:49:58 DEBUG --- stdout --- 12:49:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12990m 8619Mi am-7d597cf7d6-8vr8f 11535m 8653Mi am-7d597cf7d6-9wz9h 15770m 8980Mi ds-cts-0 11250m 8122Mi ds-cts-1 12789m 8390Mi ds-cts-2 11977m 8123Mi ds-idrepo-0 13392m 4555Mi ds-idrepo-1 1979m 4318Mi ds-idrepo-2 1968m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6464m 1557Mi 12:49:58 DEBUG --- stderr --- 12:49:58 DEBUG 12:50:05 INFO 12:50:05 INFO [loop_until]: kubectl --namespace=xlou top node 12:50:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:50:05 INFO [loop_until]: OK (rc = 0) 12:50:05 DEBUG --- stdout --- 12:50:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4869Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12086m 40% 10730Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13040m 43% 10696Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17558m 58% 3038Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2146m 7% 6608Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 10949m 36% 11997Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 55m 0% 2930Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12483m 41% 10112Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2032m 6% 5733Mi 5% gke-xlou-cdm-ds-0f601751-j488 12113m 40% 10345Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2223Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16787m 56% 10565Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13347m 44% 6232Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6823m 42% 5151Mi 8% 12:50:05 DEBUG --- stderr --- 12:50:05 DEBUG 12:50:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095442" 12:50:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095442" 12:50:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095442" 12:50:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095442" 12:50:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095442" 12:50:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095442" 12:50:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095442" 12:50:42 INFO [http_cmd]: http status code OK 12:50:42 DEBUG --- status code --- 12:50:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:42 DEBUG --- http response --- 12:50:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:42 DEBUG ----- output ----- 12:50:42 DEBUG { 12:50:42 DEBUG "status": "success", 12:50:42 DEBUG "data": { 12:50:42 DEBUG "resultType": "vector", 12:50:42 DEBUG "result": [ 12:50:42 DEBUG { 12:50:42 DEBUG "metric": { 12:50:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG [...] 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 1713095442, 12:50:42 DEBUG "2289.2666666666664" 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG } 12:50:42 ERROR [http_cmd]: ERROR 12:50:42 ERROR --- status code --- 12:50:42 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:50:42 ERROR --- http response --- 12:50:42 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:50:42 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:50:42 INFO [http_cmd]: http status code OK 12:50:42 DEBUG --- status code --- 12:50:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:42 DEBUG --- http response --- 12:50:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:50:42 DEBUG ----- output ----- 12:50:42 DEBUG { 12:50:42 DEBUG "status": "success", 12:50:42 DEBUG "data": { 12:50:42 DEBUG "resultType": "vector", 12:50:42 DEBUG "result": [ 12:50:42 DEBUG { 12:50:42 DEBUG "metric": { 12:50:42 DEBUG "instance": "10.142.0.107:9100" 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG [...] 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG "instance": "10.142.0.43:9100" 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 1713095442, 12:50:42 DEBUG "6970.333333333333" 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG } 12:50:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095442" 12:50:42 INFO [http_cmd]: http status code OK 12:50:42 DEBUG --- status code --- 12:50:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:42 DEBUG --- http response --- 12:50:42 INFO [http_cmd]: http status code OK 12:50:42 INFO [http_cmd]: http status code OK 12:50:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:50:42 DEBUG --- status code --- 12:50:42 DEBUG --- status code --- 12:50:42 DEBUG ----- output ----- 12:50:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:42 DEBUG { 12:50:42 DEBUG --- http response --- 12:50:42 DEBUG --- http response --- 12:50:42 DEBUG "status": "success", 12:50:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:50:42 DEBUG "data": { 12:50:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:50:42 DEBUG ----- output ----- 12:50:42 DEBUG "resultType": "vector", 12:50:42 DEBUG ----- output ----- 12:50:42 DEBUG { 12:50:42 DEBUG "result": [ 12:50:42 DEBUG { 12:50:42 DEBUG "status": "success", 12:50:42 DEBUG { 12:50:42 DEBUG "status": "success", 12:50:42 DEBUG "data": { 12:50:42 DEBUG "metric": { 12:50:42 DEBUG "data": { 12:50:42 DEBUG "resultType": "vector", 12:50:42 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:50:42 DEBUG "resultType": "vector", 12:50:42 DEBUG "result": [ 12:50:42 DEBUG }, 12:50:42 DEBUG "result": [ 12:50:42 DEBUG { 12:50:42 INFO [http_cmd]: http status code OK 12:50:42 DEBUG "value": [ 12:50:42 DEBUG { 12:50:42 DEBUG "metric": { 12:50:42 DEBUG --- status code --- 12:50:42 DEBUG 12:50:42 DEBUG "metric": { 12:50:42 DEBUG "pod": "ds-idrepo-1" 12:50:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:42 DEBUG 12:50:42 DEBUG "pod": "ds-idrepo-1" 12:50:42 DEBUG }, 12:50:42 DEBUG --- http response --- 12:50:42 DEBUG 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG [...] 12:50:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG ----- output ----- 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG { 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG "status": "success", 12:50:42 DEBUG 12:50:42 DEBUG [...] 12:50:42 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:50:42 DEBUG "data": { 12:50:42 DEBUG [...] 12:50:42 DEBUG 12:50:42 DEBUG }, 12:50:42 DEBUG "resultType": "vector", 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG "value": [ 12:50:42 DEBUG "result": [ 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 1713095442, 12:50:42 DEBUG { 12:50:42 DEBUG 12:50:42 DEBUG "pod": "ds-cts-1" 12:50:42 DEBUG "6.678730149248731" 12:50:42 DEBUG "metric": { 12:50:42 DEBUG "pod": "ds-idrepo-0" 12:50:42 DEBUG }, 12:50:42 DEBUG ] 12:50:42 DEBUG "pod": "ds-idrepo-1" 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG } 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 1713095442, 12:50:42 DEBUG ] 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 1713095442, 12:50:42 DEBUG "0.00006666666666666667" 12:50:42 DEBUG } 12:50:42 DEBUG 12:50:42 DEBUG "0" 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG 12:50:42 DEBUG } 12:50:42 DEBUG ] 12:50:42 DEBUG [...] 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG 12:50:42 DEBUG } 12:50:42 DEBUG } 12:50:42 DEBUG 12:50:42 DEBUG } 12:50:42 DEBUG 12:50:42 DEBUG "pod": "ds-cts-1" 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 1713095442, 12:50:42 DEBUG "0" 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG } 12:50:42 INFO [http_cmd]: http status code OK 12:50:42 DEBUG --- status code --- 12:50:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:42 DEBUG --- http response --- 12:50:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 12:50:42 DEBUG ----- output ----- 12:50:42 DEBUG { 12:50:42 DEBUG "status": "success", 12:50:42 DEBUG "data": { 12:50:42 DEBUG "resultType": "vector", 12:50:42 DEBUG "result": [ 12:50:42 DEBUG { 12:50:42 DEBUG "metric": { 12:50:42 DEBUG "pod": "ds-cts-2" 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG [...] 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 1713095442, 12:50:42 DEBUG "0" 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG } 12:50:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095442" 12:50:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095442" 12:50:42 INFO [http_cmd]: http status code OK 12:50:42 INFO [http_cmd]: http status code OK 12:50:42 DEBUG --- status code --- 12:50:42 DEBUG --- status code --- 12:50:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:42 DEBUG --- http response --- 12:50:42 DEBUG --- http response --- 12:50:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:42 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:50:42 DEBUG ----- output ----- 12:50:42 DEBUG { 12:50:42 DEBUG "status": "success", 12:50:42 DEBUG "data": { 12:50:42 DEBUG "resultType": "vector", 12:50:42 DEBUG "result": [ 12:50:42 DEBUG { 12:50:42 DEBUG "metric": { 12:50:42 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG [...] 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG 12:50:42 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:50:42 DEBUG }, 12:50:42 DEBUG "value": [ 12:50:42 DEBUG 1713095442, 12:50:42 DEBUG "2303.733333333333" 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG ] 12:50:42 DEBUG } 12:50:42 DEBUG } 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095443" 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095443" 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "instance": "10.142.0.107:9100" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "instance": "10.142.0.43:9100" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "12333.566666666664" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "instance": "10.142.0.107:9100" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "instance": "10.142.0.43:9100" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "13005775872" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "pod": "ds-idrepo-1" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "pod": "ds-idrepo-0" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "0" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "0" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095443" 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "0" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:50:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:50:43 DEBUG }, 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:50:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:50:43 DEBUG }, 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "1728483328" 12:50:43 DEBUG "2287.8333333333335" 12:50:43 DEBUG ] 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095443" 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "instance": "10.142.0.107:9100" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "instance": "10.142.0.43:9100" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "0.0004333333333332708" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "0" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:43 DEBUG { 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG { 12:50:43 DEBUG "data": { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "result": [ 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG { 12:50:43 DEBUG "result": [ 12:50:43 DEBUG "metric": { 12:50:43 DEBUG { 12:50:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:50:43 DEBUG "metric": { 12:50:43 DEBUG }, 12:50:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:50:43 DEBUG "value": [ 12:50:43 DEBUG }, 12:50:43 DEBUG 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:50:43 DEBUG 12:50:43 DEBUG }, 12:50:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:50:43 DEBUG "value": [ 12:50:43 DEBUG }, 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG "412716.5" 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG ] 12:50:43 DEBUG "0.036" 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "2288.4333333333334" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "pod": "ds-idrepo-1" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "pod": "ds-idrepo-0" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "0" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "pod": "ds-idrepo-1" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "pod": "ds-idrepo-0" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "0" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "0" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095443" 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "pod": "ds-idrepo-1" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "pod": "ds-idrepo-0" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "0" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095443" 12:50:43 INFO [http_cmd]: http status code OK 12:50:43 DEBUG --- status code --- 12:50:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:50:43 DEBUG --- http response --- 12:50:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:50:43 DEBUG ----- output ----- 12:50:43 DEBUG { 12:50:43 DEBUG "status": "success", 12:50:43 DEBUG "data": { 12:50:43 DEBUG "resultType": "vector", 12:50:43 DEBUG "result": [ 12:50:43 DEBUG { 12:50:43 DEBUG "metric": { 12:50:43 DEBUG "instance": "10.142.0.107:9100" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG [...] 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG 12:50:43 DEBUG "instance": "10.142.0.43:9100" 12:50:43 DEBUG }, 12:50:43 DEBUG "value": [ 12:50:43 DEBUG 1713095443, 12:50:43 DEBUG "0.00033333333333333365" 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG ] 12:50:43 DEBUG } 12:50:43 DEBUG } 12:50:58 INFO 12:50:58 INFO [loop_until]: kubectl --namespace=xlou top pods 12:50:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:50:59 INFO [loop_until]: OK (rc = 0) 12:50:59 DEBUG --- stdout --- 12:50:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12553m 8622Mi am-7d597cf7d6-8vr8f 11726m 8652Mi am-7d597cf7d6-9wz9h 15638m 9043Mi ds-cts-0 11282m 8117Mi ds-cts-1 11857m 8415Mi ds-cts-2 12124m 8108Mi ds-idrepo-0 13248m 4555Mi ds-idrepo-1 1996m 4317Mi ds-idrepo-2 1863m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6494m 1562Mi 12:50:59 DEBUG --- stderr --- 12:50:59 DEBUG 12:51:05 INFO 12:51:05 INFO [loop_until]: kubectl --namespace=xlou top node 12:51:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:51:05 INFO [loop_until]: OK (rc = 0) 12:51:05 DEBUG --- stdout --- 12:51:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4882Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12116m 40% 10720Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13082m 43% 10689Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17532m 58% 3062Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2124m 7% 6609Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11848m 39% 12000Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2928Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12372m 41% 10115Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1982m 6% 5721Mi 5% gke-xlou-cdm-ds-0f601751-j488 12389m 41% 10345Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2228Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16965m 56% 10620Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13325m 44% 6228Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6816m 42% 5152Mi 8% 12:51:05 DEBUG --- stderr --- 12:51:05 DEBUG 12:51:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095502" 12:51:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095502" 12:51:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095502" 12:51:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095502" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095503" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 ERROR [http_cmd]: ERROR 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 ERROR --- status code --- 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 DEBUG --- status code --- 12:51:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 ERROR --- http response --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG { 12:51:43 DEBUG { 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "data": { 12:51:43 DEBUG "data": { 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG "result": [ 12:51:43 DEBUG "result": [ 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG { 12:51:43 DEBUG { 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "ds-idrepo-1" 12:51:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:51:43 DEBUG "pod": "ds-idrepo-1" 12:51:43 DEBUG "instance": "10.142.0.107:9100" 12:51:43 DEBUG }, 12:51:43 DEBUG }, 12:51:43 DEBUG }, 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG "value": [ 12:51:43 DEBUG "value": [ 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG [...] 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "ds-idrepo-0" 12:51:43 DEBUG "pod": "ds-cts-1" 12:51:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:51:43 DEBUG 12:51:43 DEBUG }, 12:51:43 DEBUG }, 12:51:43 DEBUG }, 12:51:43 DEBUG "instance": "10.142.0.43:9100" 12:51:43 DEBUG "value": [ 12:51:43 DEBUG "value": [ 12:51:43 DEBUG "value": [ 12:51:43 DEBUG }, 12:51:43 DEBUG 1713095502, 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG 1713095502, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG "0" 12:51:43 DEBUG "0" 12:51:43 DEBUG "2299.733333333333" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG 1713095502, 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG ] 12:51:43 DEBUG ] 12:51:43 DEBUG ] 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG "5252.3" 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG ] 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG ] 12:51:43 DEBUG ] 12:51:43 DEBUG ] 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG } 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG ] 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:43 DEBUG } 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG } 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG } 12:51:43 DEBUG { 12:51:43 DEBUG { 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "data": { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "pod": "ds-idrepo-1" 12:51:43 DEBUG }, 12:51:43 DEBUG "result": [ 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG { 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG "metric": { 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG }, 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG "value": [ 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:51:43 DEBUG [...] 12:51:43 DEBUG "pod": "ds-cts-1" 12:51:43 DEBUG }, 12:51:43 DEBUG 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "2283.8333333333335" 12:51:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:51:43 DEBUG "0.00009999999999999998" 12:51:43 DEBUG ] 12:51:43 DEBUG }, 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG "value": [ 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG "6.663556232239622" 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "ds-cts-2" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "ds-idrepo-0" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "5474.472066292436" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095503" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095503" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "instance": "10.142.0.107:9100" 12:51:43 DEBUG "instance": "10.142.0.107:9100" 12:51:43 DEBUG }, 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "instance": "10.142.0.43:9100" 12:51:43 DEBUG "instance": "10.142.0.43:9100" 12:51:43 DEBUG }, 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "13006429184" 12:51:43 DEBUG "12301.3" 12:51:43 DEBUG ] 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "ds-idrepo-1" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "ds-idrepo-0" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "0" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "0" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095503" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG { 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:51:43 DEBUG 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG { 12:51:43 DEBUG 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG [...] 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG "data": { 12:51:43 DEBUG 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:51:43 DEBUG 12:51:43 DEBUG "result": [ 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG 12:51:43 DEBUG { 12:51:43 DEBUG { 12:51:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG }, 12:51:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:51:43 DEBUG "data": { 12:51:43 DEBUG "value": [ 12:51:43 DEBUG }, 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG "result": [ 12:51:43 DEBUG "2295.0333333333333" 12:51:43 DEBUG 12:51:43 DEBUG { 12:51:43 DEBUG ] 12:51:43 DEBUG 12:51:43 DEBUG "metric": { 12:51:43 DEBUG } 12:51:43 DEBUG 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095503" 12:51:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:51:43 DEBUG ] 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG }, 12:51:43 DEBUG } 12:51:43 DEBUG 12:51:43 DEBUG "value": [ 12:51:43 DEBUG } 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "ds-idrepo-0" 12:51:43 DEBUG 12:51:43 DEBUG }, 12:51:43 DEBUG 12:51:43 DEBUG "value": [ 12:51:43 DEBUG [...] 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG 12:51:43 DEBUG "13.291979216787423" 12:51:43 DEBUG 12:51:43 DEBUG ] 12:51:43 DEBUG 12:51:43 DEBUG } 12:51:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:51:43 DEBUG ] 12:51:43 DEBUG }, 12:51:43 DEBUG } 12:51:43 DEBUG "value": [ 12:51:43 DEBUG } 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "1735389184" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "instance": "10.142.0.107:9100" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "instance": "10.142.0.43:9100" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "0.0004333333333332708" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "ds-idrepo-0" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "0" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "0.037" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "412387" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "2301.0333333333333" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "ds-idrepo-0" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "159661061.74819568" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "ds-idrepo-1" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "ds-idrepo-0" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "0" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "ds-idrepo-1" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "ds-idrepo-0" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "0" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095503" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "pod": "ds-idrepo-1" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "pod": "ds-idrepo-0" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "0" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095503" 12:51:43 INFO [http_cmd]: http status code OK 12:51:43 DEBUG --- status code --- 12:51:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:51:43 DEBUG --- http response --- 12:51:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:51:43 DEBUG ----- output ----- 12:51:43 DEBUG { 12:51:43 DEBUG "status": "success", 12:51:43 DEBUG "data": { 12:51:43 DEBUG "resultType": "vector", 12:51:43 DEBUG "result": [ 12:51:43 DEBUG { 12:51:43 DEBUG "metric": { 12:51:43 DEBUG "instance": "10.142.0.107:9100" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG [...] 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG 12:51:43 DEBUG "instance": "10.142.0.43:9100" 12:51:43 DEBUG }, 12:51:43 DEBUG "value": [ 12:51:43 DEBUG 1713095503, 12:51:43 DEBUG "0" 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG ] 12:51:43 DEBUG } 12:51:43 DEBUG } 12:51:59 INFO 12:51:59 INFO [loop_until]: kubectl --namespace=xlou top pods 12:51:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:51:59 INFO [loop_until]: OK (rc = 0) 12:51:59 DEBUG --- stdout --- 12:51:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12633m 8617Mi am-7d597cf7d6-8vr8f 11857m 8652Mi am-7d597cf7d6-9wz9h 16064m 9038Mi ds-cts-0 11823m 8136Mi ds-cts-1 12119m 8384Mi ds-cts-2 12410m 8110Mi ds-idrepo-0 13515m 4555Mi ds-idrepo-1 1998m 4318Mi ds-idrepo-2 1918m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6764m 1561Mi 12:51:59 DEBUG --- stderr --- 12:51:59 DEBUG 12:52:05 INFO 12:52:05 INFO [loop_until]: kubectl --namespace=xlou top node 12:52:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:52:05 INFO [loop_until]: OK (rc = 0) 12:52:05 DEBUG --- stdout --- 12:52:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 4873Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12220m 40% 10726Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12954m 43% 10686Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17565m 58% 3044Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2139m 7% 6610Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11686m 39% 12018Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2919Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12431m 41% 10121Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1972m 6% 5735Mi 5% gke-xlou-cdm-ds-0f601751-j488 12063m 40% 10347Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2222Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16397m 54% 10622Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13360m 44% 6224Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6837m 43% 5156Mi 8% 12:52:05 DEBUG --- stderr --- 12:52:05 DEBUG 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095563" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 ERROR [http_cmd]: ERROR 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 ERROR --- status code --- 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG --- http response --- 12:52:43 ERROR --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG ----- output ----- 12:52:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:52:43 DEBUG { 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG { 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:52:43 DEBUG "metric": { 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG "pod": "ds-idrepo-1" 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "pod": "ds-idrepo-1" 12:52:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:52:43 DEBUG { 12:52:43 DEBUG }, 12:52:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:52:43 DEBUG }, 12:52:43 DEBUG }, 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "value": [ 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG "value": [ 12:52:43 DEBUG "data": { 12:52:43 DEBUG 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "result": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG { 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG [...] 12:52:43 DEBUG "metric": { 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "ds-idrepo-1" 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG }, 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "value": [ 12:52:43 DEBUG "pod": "ds-cts-1" 12:52:43 DEBUG 12:52:43 DEBUG "pod": "ds-idrepo-0" 12:52:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:52:43 DEBUG 12:52:43 DEBUG }, 12:52:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG }, 12:52:43 DEBUG }, 12:52:43 DEBUG 12:52:43 DEBUG "value": [ 12:52:43 DEBUG }, 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG "value": [ 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG [...] 12:52:43 DEBUG "0" 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG "0" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG "2306" 12:52:43 DEBUG 12:52:43 DEBUG ] 12:52:43 DEBUG "2274.2666666666664" 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:43 DEBUG ] 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG ] 12:52:43 DEBUG 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG } 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG } 12:52:43 DEBUG 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG { 12:52:43 DEBUG ] 12:52:43 DEBUG { 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG ] 12:52:43 DEBUG "pod": "ds-cts-1" 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:43 DEBUG } 12:52:43 DEBUG }, 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 DEBUG "data": { 12:52:43 DEBUG } 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG "data": { 12:52:43 DEBUG } 12:52:43 DEBUG "value": [ 12:52:43 DEBUG } 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "result": [ 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG "result": [ 12:52:43 DEBUG "0" 12:52:43 DEBUG { 12:52:43 DEBUG { 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:43 DEBUG ] 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG } 12:52:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:52:43 DEBUG "instance": "10.142.0.107:9100" 12:52:43 DEBUG { 12:52:43 DEBUG ] 12:52:43 DEBUG }, 12:52:43 DEBUG }, 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG } 12:52:43 DEBUG "value": [ 12:52:43 DEBUG "value": [ 12:52:43 DEBUG "data": { 12:52:43 DEBUG } 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "result": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG { 12:52:43 DEBUG [...] 12:52:43 DEBUG [...] 12:52:43 DEBUG "metric": { 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG }, 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "value": [ 12:52:43 DEBUG "pod": "idm-b85f9d978-rl76f" 12:52:43 DEBUG "instance": "10.142.0.43:9100" 12:52:43 DEBUG 12:52:43 DEBUG }, 12:52:43 DEBUG }, 12:52:43 DEBUG 12:52:43 DEBUG "value": [ 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG [...] 12:52:43 DEBUG "571.5282380437436" 12:52:43 DEBUG "5343.866666666667" 12:52:43 DEBUG 12:52:43 DEBUG ] 12:52:43 DEBUG ] 12:52:43 DEBUG 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 DEBUG 12:52:43 DEBUG ] 12:52:43 DEBUG ] 12:52:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 DEBUG }, 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "6.6775457210381095" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "instance": "10.142.0.107:9100" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "instance": "10.142.0.43:9100" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "12264.966666666664" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "instance": "10.142.0.107:9100" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "instance": "10.142.0.43:9100" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "13007019008" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "0" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "pod": "ds-idrepo-1" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "ds-idrepo-0" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "0" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095563" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG { 12:52:43 DEBUG --- http response --- 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG "data": { 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG "resultType": "vector", 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG "result": [ 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG { 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG "metric": { 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:52:43 DEBUG { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:43 DEBUG "result": [ 12:52:43 DEBUG }, 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "value": [ 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG "data": { 12:52:43 DEBUG { 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG 12:52:43 DEBUG { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:52:43 DEBUG 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:52:43 DEBUG "data": { 12:52:43 DEBUG }, 12:52:43 DEBUG 12:52:43 DEBUG "data": { 12:52:43 DEBUG { 12:52:43 DEBUG }, 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "value": [ 12:52:43 DEBUG [...] 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "value": [ 12:52:43 DEBUG "result": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "result": [ 12:52:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:52:43 DEBUG 12:52:43 DEBUG { 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG { 12:52:43 DEBUG }, 12:52:43 DEBUG 12:52:43 DEBUG "metric": { 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:52:43 DEBUG [...] 12:52:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:52:43 DEBUG "instance": "10.142.0.107:9100" 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG }, 12:52:43 DEBUG 12:52:43 DEBUG }, 12:52:43 DEBUG }, 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG "value": [ 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "ds-idrepo-0" 12:52:43 DEBUG "0.037" 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:52:43 DEBUG 12:52:43 DEBUG }, 12:52:43 DEBUG ] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG }, 12:52:43 DEBUG [...] 12:52:43 DEBUG "value": [ 12:52:43 DEBUG } 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG ] 12:52:43 DEBUG 12:52:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG 12:52:43 DEBUG "0" 12:52:43 DEBUG } 12:52:43 DEBUG 12:52:43 DEBUG }, 12:52:43 DEBUG "2306.5" 12:52:43 DEBUG 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG 12:52:43 DEBUG "value": [ 12:52:43 DEBUG ] 12:52:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:52:43 DEBUG } 12:52:43 DEBUG "instance": "10.142.0.43:9100" 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG } 12:52:43 DEBUG }, 12:52:43 DEBUG ] 12:52:43 DEBUG }, 12:52:43 DEBUG "1734508544" 12:52:43 DEBUG ] 12:52:43 DEBUG "value": [ 12:52:43 DEBUG } 12:52:43 DEBUG "value": [ 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG } 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 DEBUG "413542.5" 12:52:43 DEBUG "0.0004666666666667633" 12:52:43 DEBUG ] 12:52:43 DEBUG ] 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG } 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "2306.5" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "idm-b85f9d978-rl76f" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "0" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "pod": "ds-idrepo-1" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "ds-idrepo-0" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "0" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "ds-idrepo-0" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "0" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "pod": "ds-idrepo-1" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "ds-idrepo-0" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "0" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095563" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "pod": "ds-idrepo-1" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "pod": "ds-idrepo-0" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "0" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095563" 12:52:43 INFO [http_cmd]: http status code OK 12:52:43 DEBUG --- status code --- 12:52:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:43 DEBUG --- http response --- 12:52:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:43 DEBUG ----- output ----- 12:52:43 DEBUG { 12:52:43 DEBUG "status": "success", 12:52:43 DEBUG "data": { 12:52:43 DEBUG "resultType": "vector", 12:52:43 DEBUG "result": [ 12:52:43 DEBUG { 12:52:43 DEBUG "metric": { 12:52:43 DEBUG "instance": "10.142.0.107:9100" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG [...] 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG 12:52:43 DEBUG "instance": "10.142.0.43:9100" 12:52:43 DEBUG }, 12:52:43 DEBUG "value": [ 12:52:43 DEBUG 1713095563, 12:52:43 DEBUG "0" 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG ] 12:52:43 DEBUG } 12:52:43 DEBUG } 12:52:59 INFO 12:52:59 INFO [loop_until]: kubectl --namespace=xlou top pods 12:52:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:52:59 INFO [loop_until]: OK (rc = 0) 12:52:59 DEBUG --- stdout --- 12:52:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12823m 8657Mi am-7d597cf7d6-8vr8f 11459m 8661Mi am-7d597cf7d6-9wz9h 15597m 8985Mi ds-cts-0 11461m 8122Mi ds-cts-1 12003m 8388Mi ds-cts-2 12269m 8137Mi ds-idrepo-0 13348m 4555Mi ds-idrepo-1 1978m 4318Mi ds-idrepo-2 1874m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6813m 1560Mi 12:52:59 DEBUG --- stderr --- 12:52:59 DEBUG 12:53:05 INFO 12:53:05 INFO [loop_until]: kubectl --namespace=xlou top node 12:53:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:53:05 INFO [loop_until]: OK (rc = 0) 12:53:05 DEBUG --- stdout --- 12:53:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4863Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12276m 41% 10738Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13033m 43% 10722Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17623m 59% 3034Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2114m 7% 6601Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11618m 38% 12000Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2925Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12404m 41% 10120Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1989m 6% 5740Mi 5% gke-xlou-cdm-ds-0f601751-j488 12333m 41% 10350Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2223Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16916m 56% 10561Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13384m 44% 6236Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6855m 43% 5155Mi 8% 12:53:05 DEBUG --- stderr --- 12:53:05 DEBUG 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "2298.233333333333" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "instance": "10.142.0.107:9100" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "instance": "10.142.0.43:9100" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "5578.599999999999" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 ERROR [http_cmd]: ERROR 12:53:43 DEBUG --- status code --- 12:53:43 ERROR --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 ERROR --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:53:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:53:43 DEBUG ----- output ----- 12:53:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "ds-idrepo-1" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-cts-1" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "0" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-idrepo-0" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "3000.6423070160013" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG { 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:53:43 DEBUG "data": { 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG "resultType": "vector", 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "result": [ 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG { 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:43 DEBUG "result": [ 12:53:43 DEBUG }, 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG "result": [ 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "value": [ 12:53:43 DEBUG { 12:53:43 DEBUG { 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "pod": "ds-idrepo-1" 12:53:43 DEBUG 12:53:43 DEBUG "data": { 12:53:43 DEBUG "pod": "ds-idrepo-1" 12:53:43 DEBUG "data": { 12:53:43 DEBUG }, 12:53:43 DEBUG 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG }, 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "value": [ 12:53:43 DEBUG [...] 12:53:43 DEBUG "result": [ 12:53:43 DEBUG "value": [ 12:53:43 DEBUG "result": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG { 12:53:43 DEBUG 12:53:43 DEBUG { 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "metric": { 12:53:43 DEBUG 12:53:43 DEBUG "metric": { 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:53:43 DEBUG 12:53:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:53:43 DEBUG [...] 12:53:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:53:43 DEBUG }, 12:53:43 DEBUG [...] 12:53:43 DEBUG }, 12:53:43 DEBUG 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-cts-1" 12:53:43 DEBUG "2288.8" 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-idrepo-0" 12:53:43 DEBUG 12:53:43 DEBUG }, 12:53:43 DEBUG ] 12:53:43 DEBUG [...] 12:53:43 DEBUG }, 12:53:43 DEBUG [...] 12:53:43 DEBUG "value": [ 12:53:43 DEBUG } 12:53:43 DEBUG 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG ] 12:53:43 DEBUG 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG 12:53:43 DEBUG "0.00027745833333333333" 12:53:43 DEBUG } 12:53:43 DEBUG 12:53:43 DEBUG "0" 12:53:43 DEBUG 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:53:43 DEBUG ] 12:53:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:53:43 DEBUG } 12:53:43 DEBUG }, 12:53:43 DEBUG } 12:53:43 DEBUG }, 12:53:43 DEBUG ] 12:53:43 DEBUG "value": [ 12:53:43 DEBUG ] 12:53:43 DEBUG "value": [ 12:53:43 DEBUG } 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG } 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG } 12:53:43 DEBUG "0" 12:53:43 DEBUG } 12:53:43 DEBUG "6.703671964123123" 12:53:43 DEBUG ] 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "instance": "10.142.0.107:9100" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "instance": "10.142.0.43:9100" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "12358.866666666665" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "instance": "10.142.0.107:9100" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "instance": "10.142.0.43:9100" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "13007744000" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "ds-idrepo-1" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-idrepo-0" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "0" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 DEBUG --- status code --- 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095623" 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "instance": "10.142.0.107:9100" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "instance": "10.142.0.43:9100" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "0.0004333333333332708" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "0.037" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "2293.3333333333335" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-idrepo-0" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "8.526071530455454" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "413656.5" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "2298.5666666666666" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "1732206592" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-idrepo-0" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "0" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "ds-idrepo-1" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-idrepo-0" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "0" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-idrepo-0" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "10040.610796553543" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "ds-idrepo-1" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-idrepo-0" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "0" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "pod": "ds-idrepo-1" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "pod": "ds-idrepo-0" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "0" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095623" 12:53:43 INFO [http_cmd]: http status code OK 12:53:43 DEBUG --- status code --- 12:53:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:43 DEBUG --- http response --- 12:53:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:43 DEBUG ----- output ----- 12:53:43 DEBUG { 12:53:43 DEBUG "status": "success", 12:53:43 DEBUG "data": { 12:53:43 DEBUG "resultType": "vector", 12:53:43 DEBUG "result": [ 12:53:43 DEBUG { 12:53:43 DEBUG "metric": { 12:53:43 DEBUG "instance": "10.142.0.107:9100" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG [...] 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG 12:53:43 DEBUG "instance": "10.142.0.43:9100" 12:53:43 DEBUG }, 12:53:43 DEBUG "value": [ 12:53:43 DEBUG 1713095623, 12:53:43 DEBUG "0.0006666666666666598" 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG ] 12:53:43 DEBUG } 12:53:43 DEBUG } 12:53:59 INFO 12:53:59 INFO [loop_until]: kubectl --namespace=xlou top pods 12:53:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:53:59 INFO [loop_until]: OK (rc = 0) 12:53:59 DEBUG --- stdout --- 12:53:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12653m 8620Mi am-7d597cf7d6-8vr8f 11720m 8650Mi am-7d597cf7d6-9wz9h 15860m 9005Mi ds-cts-0 11154m 8137Mi ds-cts-1 12065m 8383Mi ds-cts-2 12332m 8112Mi ds-idrepo-0 13046m 4555Mi ds-idrepo-1 2013m 4318Mi ds-idrepo-2 1908m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6574m 1561Mi 12:53:59 DEBUG --- stderr --- 12:53:59 DEBUG 12:54:05 INFO 12:54:05 INFO [loop_until]: kubectl --namespace=xlou top node 12:54:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:54:05 INFO [loop_until]: OK (rc = 0) 12:54:05 DEBUG --- stdout --- 12:54:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 4868Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11648m 39% 10739Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13038m 43% 10692Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17613m 58% 3046Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2138m 7% 6598Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11650m 39% 12010Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 55m 0% 2928Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12340m 41% 10110Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1985m 6% 5747Mi 5% gke-xlou-cdm-ds-0f601751-j488 12154m 40% 10344Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2224Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16988m 56% 10577Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13417m 44% 6242Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6840m 43% 5154Mi 8% 12:54:05 DEBUG --- stderr --- 12:54:05 DEBUG 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "ds-idrepo-1" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "ds-idrepo-0" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:43 ERROR [http_cmd]: ERROR 12:54:43 DEBUG ----- output ----- 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG ----- output ----- 12:54:43 ERROR --- status code --- 12:54:43 DEBUG { 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG { 12:54:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:54:43 DEBUG "status": "success", 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG "status": "success", 12:54:43 ERROR --- http response --- 12:54:43 DEBUG "data": { 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:43 DEBUG "result": [ 12:54:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG { 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG "pod": "ds-idrepo-1" 12:54:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:54:43 DEBUG { 12:54:43 DEBUG }, 12:54:43 DEBUG }, 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "data": { 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "result": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG { 12:54:43 DEBUG [...] 12:54:43 DEBUG [...] 12:54:43 DEBUG "metric": { 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "instance": "10.142.0.107:9100" 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG }, 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "pod": "ds-cts-1" 12:54:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:54:43 DEBUG 12:54:43 DEBUG }, 12:54:43 DEBUG }, 12:54:43 DEBUG 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG [...] 12:54:43 DEBUG "0" 12:54:43 DEBUG "2300.5666666666666" 12:54:43 DEBUG 12:54:43 DEBUG ] 12:54:43 DEBUG ] 12:54:43 DEBUG 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 DEBUG 12:54:43 DEBUG ] 12:54:43 DEBUG ] 12:54:43 DEBUG "instance": "10.142.0.43:9100" 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 DEBUG }, 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "7055.766666666666" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "2298.4333333333334" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "ds-idrepo-0" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "4710.830704521557" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "ds-idrepo-1" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "ds-cts-1" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0.0002" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "6.700869595381313" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095683" 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "instance": "10.142.0.107:9100" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "instance": "10.142.0.43:9100" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "11977.599999999999" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "instance": "10.142.0.107:9100" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "instance": "10.142.0.43:9100" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "13008454656" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "ds-idrepo-1" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "ds-idrepo-0" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095683" 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0.037" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "instance": "10.142.0.107:9100" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "instance": "10.142.0.43:9100" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0.00040000000000001514" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG } 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:54:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:54:43 DEBUG }, 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG "pod": "ds-idrepo-0" 12:54:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:54:43 DEBUG { 12:54:43 DEBUG }, 12:54:43 DEBUG }, 12:54:43 DEBUG "status": "success", 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "data": { 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG "13.317366160620335" 12:54:43 DEBUG "2301.733333333333" 12:54:43 DEBUG "result": [ 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG ] 12:54:43 DEBUG ] 12:54:43 DEBUG { 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 DEBUG "metric": { 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG ] 12:54:43 DEBUG ] 12:54:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:54:43 DEBUG { 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 DEBUG }, 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "data": { 12:54:43 DEBUG 12:54:43 DEBUG "resultType": "vector", 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG 12:54:43 DEBUG "result": [ 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG 12:54:43 DEBUG { 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG [...] 12:54:43 DEBUG "metric": { 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG 12:54:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:54:43 DEBUG 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:43 DEBUG }, 12:54:43 DEBUG 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:54:43 DEBUG { 12:54:43 DEBUG 12:54:43 DEBUG }, 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "data": { 12:54:43 DEBUG 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG [...] 12:54:43 DEBUG "2300.4333333333334" 12:54:43 DEBUG "result": [ 12:54:43 DEBUG 12:54:43 DEBUG ] 12:54:43 DEBUG { 12:54:43 DEBUG 12:54:43 DEBUG } 12:54:43 DEBUG "metric": { 12:54:43 DEBUG 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 DEBUG ] 12:54:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:54:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:54:43 DEBUG } 12:54:43 DEBUG }, 12:54:43 DEBUG }, 12:54:43 DEBUG } 12:54:43 DEBUG "value": [ 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG 12:54:43 DEBUG "1732907008" 12:54:43 DEBUG 12:54:43 DEBUG ] 12:54:43 DEBUG [...] 12:54:43 DEBUG } 12:54:43 DEBUG 12:54:43 DEBUG ] 12:54:43 DEBUG 12:54:43 DEBUG } 12:54:43 DEBUG 12:54:43 DEBUG } 12:54:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "413780" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "ds-idrepo-0" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "ds-idrepo-1" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "ds-idrepo-0" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "ds-idrepo-0" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "157037022.08201894" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "ds-idrepo-1" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "ds-idrepo-0" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095683" 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "pod": "ds-idrepo-1" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "pod": "ds-idrepo-0" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095683" 12:54:43 INFO [http_cmd]: http status code OK 12:54:43 DEBUG --- status code --- 12:54:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:43 DEBUG --- http response --- 12:54:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:43 DEBUG ----- output ----- 12:54:43 DEBUG { 12:54:43 DEBUG "status": "success", 12:54:43 DEBUG "data": { 12:54:43 DEBUG "resultType": "vector", 12:54:43 DEBUG "result": [ 12:54:43 DEBUG { 12:54:43 DEBUG "metric": { 12:54:43 DEBUG "instance": "10.142.0.107:9100" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG [...] 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG 12:54:43 DEBUG "instance": "10.142.0.43:9100" 12:54:43 DEBUG }, 12:54:43 DEBUG "value": [ 12:54:43 DEBUG 1713095683, 12:54:43 DEBUG "0" 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG ] 12:54:43 DEBUG } 12:54:43 DEBUG } 12:54:59 INFO 12:54:59 INFO [loop_until]: kubectl --namespace=xlou top pods 12:54:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:54:59 INFO [loop_until]: OK (rc = 0) 12:54:59 DEBUG --- stdout --- 12:54:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12647m 8624Mi am-7d597cf7d6-8vr8f 11866m 8649Mi am-7d597cf7d6-9wz9h 16090m 8990Mi ds-cts-0 11449m 8120Mi ds-cts-1 11875m 8393Mi ds-cts-2 12000m 8138Mi ds-idrepo-0 13517m 4555Mi ds-idrepo-1 2002m 4318Mi ds-idrepo-2 1922m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6570m 1561Mi 12:54:59 DEBUG --- stderr --- 12:54:59 DEBUG 12:55:05 INFO 12:55:05 INFO [loop_until]: kubectl --namespace=xlou top node 12:55:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:55:05 INFO [loop_until]: OK (rc = 0) 12:55:05 DEBUG --- stdout --- 12:55:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4880Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12168m 40% 10745Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13059m 43% 10692Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17532m 58% 3043Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2099m 7% 6610Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11735m 39% 12015Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2927Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12396m 41% 10116Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2036m 6% 5741Mi 5% gke-xlou-cdm-ds-0f601751-j488 12213m 40% 10358Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2218Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16850m 56% 10554Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13347m 44% 6241Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6809m 42% 5156Mi 8% 12:55:05 DEBUG --- stderr --- 12:55:05 DEBUG 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 ERROR [http_cmd]: ERROR 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 ERROR --- status code --- 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG --- status code --- 12:55:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 ERROR --- http response --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:55:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG { 12:55:43 DEBUG { 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "data": { 12:55:43 DEBUG "data": { 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG "result": [ 12:55:43 DEBUG "result": [ 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG { 12:55:43 DEBUG { 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:43 DEBUG "pod": "ds-idrepo-1" 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG "pod": "ds-idrepo-1" 12:55:43 DEBUG "pod": "ds-idrepo-1" 12:55:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG }, 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG }, 12:55:43 DEBUG }, 12:55:43 DEBUG }, 12:55:43 DEBUG { 12:55:43 DEBUG "value": [ 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG "value": [ 12:55:43 DEBUG "value": [ 12:55:43 DEBUG "value": [ 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "data": { 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "result": [ 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG [...] 12:55:43 DEBUG [...] 12:55:43 DEBUG { 12:55:43 DEBUG [...] 12:55:43 DEBUG [...] 12:55:43 DEBUG { 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "metric": { 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "data": { 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "instance": "10.142.0.107:9100" 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG }, 12:55:43 DEBUG "pod": "ds-cts-1" 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:43 DEBUG "pod": "ds-idrepo-0" 12:55:43 DEBUG "result": [ 12:55:43 DEBUG "pod": "ds-cts-1" 12:55:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:55:43 DEBUG "value": [ 12:55:43 DEBUG }, 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG }, 12:55:43 DEBUG { 12:55:43 DEBUG }, 12:55:43 DEBUG }, 12:55:43 DEBUG 12:55:43 DEBUG "value": [ 12:55:43 DEBUG { 12:55:43 DEBUG "value": [ 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "value": [ 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG 1713095743, 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 DEBUG 12:55:43 DEBUG "0.00016666666666666666" 12:55:43 DEBUG "data": { 12:55:43 DEBUG "0" 12:55:43 DEBUG }, 12:55:43 DEBUG "0" 12:55:43 DEBUG "2283.7" 12:55:43 DEBUG [...] 12:55:43 DEBUG ] 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG ] 12:55:43 DEBUG "value": [ 12:55:43 DEBUG ] 12:55:43 DEBUG ] 12:55:43 DEBUG 12:55:43 DEBUG } 12:55:43 DEBUG "result": [ 12:55:43 DEBUG } 12:55:43 DEBUG 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 DEBUG 12:55:43 DEBUG ] 12:55:43 DEBUG { 12:55:43 DEBUG ] 12:55:43 DEBUG 12:55:43 DEBUG ] 12:55:43 DEBUG ] 12:55:43 DEBUG 12:55:43 DEBUG } 12:55:43 DEBUG "metric": { 12:55:43 DEBUG } 12:55:43 DEBUG 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 DEBUG "instance": "10.142.0.43:9100" 12:55:43 DEBUG } 12:55:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:55:43 DEBUG } 12:55:43 DEBUG [...] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 DEBUG }, 12:55:43 DEBUG }, 12:55:43 DEBUG 12:55:43 DEBUG "value": [ 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "6629.2" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 DEBUG 12:55:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:55:43 DEBUG ] 12:55:43 DEBUG 12:55:43 DEBUG }, 12:55:43 DEBUG } 12:55:43 DEBUG [...] 12:55:43 DEBUG "value": [ 12:55:43 DEBUG ] 12:55:43 DEBUG 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG } 12:55:43 DEBUG 12:55:43 DEBUG "6.677827545814281" 12:55:43 DEBUG } 12:55:43 DEBUG 12:55:43 DEBUG ] 12:55:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:55:43 DEBUG } 12:55:43 DEBUG }, 12:55:43 DEBUG ] 12:55:43 DEBUG "value": [ 12:55:43 DEBUG } 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG } 12:55:43 DEBUG "2307" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "0" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "pod": "ds-idrepo-0" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "3469.5014835437246" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095743" 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "instance": "10.142.0.107:9100" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "instance": "10.142.0.43:9100" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "13008995328" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "instance": "10.142.0.107:9100" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "instance": "10.142.0.43:9100" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "11834.199999999997" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "ds-idrepo-1" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "pod": "ds-idrepo-0" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "0" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095743" 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "0.037" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095743" 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "instance": "10.142.0.107:9100" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "instance": "10.142.0.43:9100" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "0.000333333333333267" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "2287.1" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG --- status code --- 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:55:43 DEBUG }, 12:55:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "pod": "ds-idrepo-0" 12:55:43 DEBUG 12:55:43 DEBUG "pod": "ds-idrepo-0" 12:55:43 DEBUG }, 12:55:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "9.282182855580208" 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "12576.942877846002" 12:55:43 DEBUG ] 12:55:43 DEBUG "2283.9333333333334" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "ds-idrepo-1" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "pod": "ds-idrepo-0" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "0" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "pod": "ds-idrepo-0" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "0" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG { 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:43 DEBUG "data": { 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG { 12:55:43 DEBUG "result": [ 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG { 12:55:43 DEBUG "data": { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:55:43 DEBUG "result": [ 12:55:43 DEBUG }, 12:55:43 DEBUG { 12:55:43 DEBUG "value": [ 12:55:43 DEBUG "metric": { 12:55:43 DEBUG 12:55:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:55:43 DEBUG 12:55:43 DEBUG }, 12:55:43 DEBUG 12:55:43 DEBUG "value": [ 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:55:43 DEBUG 12:55:43 DEBUG }, 12:55:43 DEBUG 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:55:43 DEBUG "412977" 12:55:43 DEBUG }, 12:55:43 DEBUG ] 12:55:43 DEBUG "value": [ 12:55:43 DEBUG } 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG ] 12:55:43 DEBUG "1733087232" 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "ds-idrepo-1" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "pod": "ds-idrepo-0" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "0" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095743" 12:55:43 INFO [http_cmd]: http status code OK 12:55:43 DEBUG --- status code --- 12:55:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:43 DEBUG --- http response --- 12:55:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:43 DEBUG ----- output ----- 12:55:43 DEBUG { 12:55:43 DEBUG "status": "success", 12:55:43 DEBUG "data": { 12:55:43 DEBUG "resultType": "vector", 12:55:43 DEBUG "result": [ 12:55:43 DEBUG { 12:55:43 DEBUG "metric": { 12:55:43 DEBUG "pod": "ds-idrepo-1" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG [...] 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG 12:55:43 DEBUG "pod": "ds-idrepo-0" 12:55:43 DEBUG }, 12:55:43 DEBUG "value": [ 12:55:43 DEBUG 1713095743, 12:55:43 DEBUG "0" 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG ] 12:55:43 DEBUG } 12:55:43 DEBUG } 12:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095744" 12:55:44 INFO [http_cmd]: http status code OK 12:55:44 DEBUG --- status code --- 12:55:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:44 DEBUG --- http response --- 12:55:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:44 DEBUG ----- output ----- 12:55:44 DEBUG { 12:55:44 DEBUG "status": "success", 12:55:44 DEBUG "data": { 12:55:44 DEBUG "resultType": "vector", 12:55:44 DEBUG "result": [ 12:55:44 DEBUG { 12:55:44 DEBUG "metric": { 12:55:44 DEBUG "instance": "10.142.0.107:9100" 12:55:44 DEBUG }, 12:55:44 DEBUG "value": [ 12:55:44 DEBUG 12:55:44 DEBUG 12:55:44 DEBUG 12:55:44 DEBUG [...] 12:55:44 DEBUG 12:55:44 DEBUG 12:55:44 DEBUG 12:55:44 DEBUG "instance": "10.142.0.43:9100" 12:55:44 DEBUG }, 12:55:44 DEBUG "value": [ 12:55:44 DEBUG 1713095744, 12:55:44 DEBUG "0" 12:55:44 DEBUG ] 12:55:44 DEBUG } 12:55:44 DEBUG ] 12:55:44 DEBUG } 12:55:44 DEBUG } 12:55:59 INFO 12:55:59 INFO [loop_until]: kubectl --namespace=xlou top pods 12:55:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:55:59 INFO [loop_until]: OK (rc = 0) 12:55:59 DEBUG --- stdout --- 12:55:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12713m 8622Mi am-7d597cf7d6-8vr8f 11773m 8651Mi am-7d597cf7d6-9wz9h 15938m 9032Mi ds-cts-0 11663m 8146Mi ds-cts-1 11892m 8394Mi ds-cts-2 11980m 8138Mi ds-idrepo-0 13534m 4555Mi ds-idrepo-1 2049m 4318Mi ds-idrepo-2 1882m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6539m 1561Mi 12:55:59 DEBUG --- stderr --- 12:55:59 DEBUG 12:56:05 INFO 12:56:05 INFO [loop_until]: kubectl --namespace=xlou top node 12:56:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:56:05 INFO [loop_until]: OK (rc = 0) 12:56:05 DEBUG --- stdout --- 12:56:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4876Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12096m 40% 10745Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13039m 43% 10697Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17599m 58% 3044Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2137m 7% 6602Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11843m 39% 12013Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2938Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12521m 41% 10116Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2034m 6% 5746Mi 5% gke-xlou-cdm-ds-0f601751-j488 12206m 40% 10357Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2230Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16988m 56% 10602Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13427m 44% 6234Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6832m 42% 5156Mi 8% 12:56:05 DEBUG --- stderr --- 12:56:05 DEBUG 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095803" 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "ds-idrepo-1" 12:56:43 DEBUG }, 12:56:43 DEBUG "instance": "10.142.0.107:9100" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:56:43 DEBUG 12:56:43 DEBUG "pod": "ds-cts-1" 12:56:43 DEBUG }, 12:56:43 DEBUG "instance": "10.142.0.43:9100" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "2301.7" 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0" 12:56:43 DEBUG ] 12:56:43 DEBUG "5301.666666666666" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095803" 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "ds-idrepo-1" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "ds-cts-1" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0.000554625" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "6.695345144020916" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 ERROR [http_cmd]: ERROR 12:56:43 ERROR --- status code --- 12:56:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:56:43 ERROR --- http response --- 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:56:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "2299" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG { 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "ds-idrepo-1" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "ds-idrepo-0" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095803" 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG { 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "instance": "10.142.0.107:9100" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "instance": "10.142.0.43:9100" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "13009619968" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "instance": "10.142.0.107:9100" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "instance": "10.142.0.43:9100" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "12212.533333333331" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "ds-idrepo-1" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "ds-idrepo-0" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "109761599.10115495" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0.037" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "413040.5" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "13.279834196335209" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "instance": "10.142.0.107:9100" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "instance": "10.142.0.43:9100" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0.00036666666666675953" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "2301.766666666667" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095803" 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "2301.7" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG { 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG "data": { 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:56:43 DEBUG { 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG "metric": { 12:56:43 DEBUG { 12:56:43 DEBUG "pod": "ds-idrepo-1" 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG }, 12:56:43 DEBUG "data": { 12:56:43 DEBUG "value": [ 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG 12:56:43 DEBUG "result": [ 12:56:43 DEBUG 12:56:43 DEBUG { 12:56:43 DEBUG 12:56:43 DEBUG "metric": { 12:56:43 DEBUG [...] 12:56:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:56:43 DEBUG 12:56:43 DEBUG }, 12:56:43 DEBUG 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "ds-idrepo-0" 12:56:43 DEBUG 12:56:43 DEBUG }, 12:56:43 DEBUG 12:56:43 DEBUG "value": [ 12:56:43 DEBUG [...] 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG 12:56:43 DEBUG "0" 12:56:43 DEBUG 12:56:43 DEBUG ] 12:56:43 DEBUG 12:56:43 DEBUG } 12:56:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 12:56:43 DEBUG ] 12:56:43 DEBUG }, 12:56:43 DEBUG } 12:56:43 DEBUG "value": [ 12:56:43 DEBUG } 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "1733931008" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "ds-idrepo-1" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "ds-idrepo-0" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095803" 12:56:43 INFO [http_cmd]: http status code OK 12:56:43 DEBUG --- status code --- 12:56:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:43 DEBUG --- http response --- 12:56:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:43 DEBUG ----- output ----- 12:56:43 DEBUG { 12:56:43 DEBUG "status": "success", 12:56:43 DEBUG "data": { 12:56:43 DEBUG "resultType": "vector", 12:56:43 DEBUG "result": [ 12:56:43 DEBUG { 12:56:43 DEBUG "metric": { 12:56:43 DEBUG "pod": "ds-idrepo-1" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG [...] 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG 12:56:43 DEBUG "pod": "ds-idrepo-0" 12:56:43 DEBUG }, 12:56:43 DEBUG "value": [ 12:56:43 DEBUG 1713095803, 12:56:43 DEBUG "0" 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG ] 12:56:43 DEBUG } 12:56:43 DEBUG } 12:56:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095804" 12:56:44 INFO [http_cmd]: http status code OK 12:56:44 DEBUG --- status code --- 12:56:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:44 DEBUG --- http response --- 12:56:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:44 DEBUG ----- output ----- 12:56:44 DEBUG { 12:56:44 DEBUG "status": "success", 12:56:44 DEBUG "data": { 12:56:44 DEBUG "resultType": "vector", 12:56:44 DEBUG "result": [ 12:56:44 DEBUG { 12:56:44 DEBUG "metric": { 12:56:44 DEBUG "instance": "10.142.0.107:9100" 12:56:44 DEBUG }, 12:56:44 DEBUG "value": [ 12:56:44 DEBUG 12:56:44 DEBUG 12:56:44 DEBUG 12:56:44 DEBUG [...] 12:56:44 DEBUG 12:56:44 DEBUG 12:56:44 DEBUG 12:56:44 DEBUG "instance": "10.142.0.43:9100" 12:56:44 DEBUG }, 12:56:44 DEBUG "value": [ 12:56:44 DEBUG 1713095804, 12:56:44 DEBUG "0.00033333333333333365" 12:56:44 DEBUG ] 12:56:44 DEBUG } 12:56:44 DEBUG ] 12:56:44 DEBUG } 12:56:44 DEBUG } 12:56:59 INFO 12:56:59 INFO [loop_until]: kubectl --namespace=xlou top pods 12:56:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:56:59 INFO [loop_until]: OK (rc = 0) 12:56:59 DEBUG --- stdout --- 12:56:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12801m 8676Mi am-7d597cf7d6-8vr8f 11488m 8651Mi am-7d597cf7d6-9wz9h 16512m 9034Mi ds-cts-0 11640m 8126Mi ds-cts-1 12255m 8419Mi ds-cts-2 12429m 8166Mi ds-idrepo-0 13354m 4555Mi ds-idrepo-1 1977m 4318Mi ds-idrepo-2 1873m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6741m 1561Mi 12:56:59 DEBUG --- stderr --- 12:56:59 DEBUG 12:57:06 INFO 12:57:06 INFO [loop_until]: kubectl --namespace=xlou top node 12:57:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:57:06 INFO [loop_until]: OK (rc = 0) 12:57:06 DEBUG --- stdout --- 12:57:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4884Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12006m 40% 10792Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13198m 44% 10736Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17613m 58% 3064Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2085m 6% 6592Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11292m 37% 12004Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2933Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12478m 41% 10109Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1979m 6% 5748Mi 5% gke-xlou-cdm-ds-0f601751-j488 12595m 42% 10355Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2213Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 17024m 57% 10603Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13400m 44% 6222Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6829m 42% 5160Mi 8% 12:57:06 DEBUG --- stderr --- 12:57:06 DEBUG 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "ds-idrepo-1" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "ds-cts-1" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "0" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "6.69456389871496" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG { 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG "metric": { 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG "pod": "ds-idrepo-1" 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:43 DEBUG 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG 12:57:43 DEBUG { 12:57:43 DEBUG "pod": "ds-cts-1" 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG }, 12:57:43 DEBUG "data": { 12:57:43 DEBUG "value": [ 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "result": [ 12:57:43 DEBUG "0.0010514583333333333" 12:57:43 DEBUG { 12:57:43 DEBUG ] 12:57:43 DEBUG "metric": { 12:57:43 DEBUG } 12:57:43 DEBUG "instance": "10.142.0.107:9100" 12:57:43 DEBUG ] 12:57:43 DEBUG }, 12:57:43 DEBUG } 12:57:43 DEBUG "value": [ 12:57:43 DEBUG } 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "instance": "10.142.0.43:9100" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "5248.533333333333" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "2296.0333333333333" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 ERROR [http_cmd]: ERROR 12:57:43 ERROR --- status code --- 12:57:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:57:43 ERROR --- http response --- 12:57:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:57:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:57:43 DEBUG "pod": "ds-idrepo-1" 12:57:43 DEBUG }, 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:57:43 DEBUG "pod": "ds-idrepo-0" 12:57:43 DEBUG }, 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "2302.2" 12:57:43 DEBUG "0" 12:57:43 DEBUG ] 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "0" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "5230551.259183118" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "instance": "10.142.0.107:9100" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "instance": "10.142.0.43:9100" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "12155.766666666665" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "instance": "10.142.0.107:9100" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "instance": "10.142.0.43:9100" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "13010267136" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "ds-idrepo-1" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "ds-idrepo-0" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "0" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "ds-idrepo-0" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "0" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "ds-idrepo-0" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "0" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG { 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:43 DEBUG "data": { 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG }, 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "value": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "instance": "10.142.0.107:9100" 12:57:43 DEBUG 12:57:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:57:43 DEBUG }, 12:57:43 DEBUG 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG [...] 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG }, 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG 12:57:43 DEBUG "instance": "10.142.0.43:9100" 12:57:43 DEBUG "413257.5" 12:57:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:57:43 DEBUG }, 12:57:43 DEBUG ] 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG } 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG ] 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "0.0004666666666667633" 12:57:43 DEBUG } 12:57:43 DEBUG "0.037" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "2297.9" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "2295.9666666666667" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "ds-idrepo-1" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "ds-idrepo-0" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "0" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "0" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "1734131712" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "ds-idrepo-1" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "ds-idrepo-0" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "0" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095863" 12:57:43 INFO [http_cmd]: http status code OK 12:57:43 DEBUG --- status code --- 12:57:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:43 DEBUG --- http response --- 12:57:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:43 DEBUG ----- output ----- 12:57:43 DEBUG { 12:57:43 DEBUG "status": "success", 12:57:43 DEBUG "data": { 12:57:43 DEBUG "resultType": "vector", 12:57:43 DEBUG "result": [ 12:57:43 DEBUG { 12:57:43 DEBUG "metric": { 12:57:43 DEBUG "pod": "ds-idrepo-1" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG [...] 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG 12:57:43 DEBUG "pod": "ds-idrepo-0" 12:57:43 DEBUG }, 12:57:43 DEBUG "value": [ 12:57:43 DEBUG 1713095863, 12:57:43 DEBUG "0" 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG ] 12:57:43 DEBUG } 12:57:43 DEBUG } 12:57:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095864" 12:57:44 INFO [http_cmd]: http status code OK 12:57:44 DEBUG --- status code --- 12:57:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:44 DEBUG --- http response --- 12:57:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:44 DEBUG ----- output ----- 12:57:44 DEBUG { 12:57:44 DEBUG "status": "success", 12:57:44 DEBUG "data": { 12:57:44 DEBUG "resultType": "vector", 12:57:44 DEBUG "result": [ 12:57:44 DEBUG { 12:57:44 DEBUG "metric": { 12:57:44 DEBUG "instance": "10.142.0.107:9100" 12:57:44 DEBUG }, 12:57:44 DEBUG "value": [ 12:57:44 DEBUG 12:57:44 DEBUG 12:57:44 DEBUG 12:57:44 DEBUG [...] 12:57:44 DEBUG 12:57:44 DEBUG 12:57:44 DEBUG 12:57:44 DEBUG "instance": "10.142.0.43:9100" 12:57:44 DEBUG }, 12:57:44 DEBUG "value": [ 12:57:44 DEBUG 1713095864, 12:57:44 DEBUG "0" 12:57:44 DEBUG ] 12:57:44 DEBUG } 12:57:44 DEBUG ] 12:57:44 DEBUG } 12:57:44 DEBUG } 12:57:59 INFO 12:57:59 INFO [loop_until]: kubectl --namespace=xlou top pods 12:57:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:57:59 INFO [loop_until]: OK (rc = 0) 12:57:59 DEBUG --- stdout --- 12:57:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12546m 8630Mi am-7d597cf7d6-8vr8f 11597m 8653Mi am-7d597cf7d6-9wz9h 15765m 8985Mi ds-cts-0 11792m 8120Mi ds-cts-1 11794m 8398Mi ds-cts-2 12415m 8205Mi ds-idrepo-0 13306m 4555Mi ds-idrepo-1 1980m 4318Mi ds-idrepo-2 1969m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6763m 1561Mi 12:57:59 DEBUG --- stderr --- 12:57:59 DEBUG 12:58:06 INFO 12:58:06 INFO [loop_until]: kubectl --namespace=xlou top node 12:58:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:58:06 INFO [loop_until]: OK (rc = 0) 12:58:06 DEBUG --- stdout --- 12:58:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4867Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12144m 40% 10824Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13082m 43% 10700Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17498m 58% 3048Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2084m 6% 6606Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11618m 38% 12022Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2933Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12371m 41% 10117Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2011m 6% 5736Mi 5% gke-xlou-cdm-ds-0f601751-j488 11782m 39% 10362Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2217Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16913m 56% 10573Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13354m 44% 6238Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6829m 42% 5157Mi 8% 12:58:06 DEBUG --- stderr --- 12:58:06 DEBUG 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG { 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:43 DEBUG "data": { 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG }, 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "value": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "instance": "10.142.0.107:9100" 12:58:43 DEBUG 12:58:43 DEBUG "pod": "ds-idrepo-1" 12:58:43 DEBUG }, 12:58:43 DEBUG 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG [...] 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG }, 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG 12:58:43 DEBUG "instance": "10.142.0.43:9100" 12:58:43 DEBUG "2283.8333333333335" 12:58:43 DEBUG "pod": "ds-cts-1" 12:58:43 DEBUG }, 12:58:43 DEBUG ] 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG } 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG ] 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "5365.533333333333" 12:58:43 DEBUG } 12:58:43 DEBUG "0.00013333333333333334" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "6.66246761498018" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "ds-idrepo-1" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "ds-cts-1" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 ERROR [http_cmd]: ERROR 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 ERROR --- status code --- 12:58:43 DEBUG --- status code --- 12:58:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 ERROR --- http response --- 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:58:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "ds-idrepo-1" 12:58:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:58:43 DEBUG }, 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "ds-idrepo-0" 12:58:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:58:43 DEBUG }, 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0" 12:58:43 DEBUG "2281.5666666666666" 12:58:43 DEBUG ] 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095923" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "ds-idrepo-2" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "3430.2274533474174" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "instance": "10.142.0.107:9100" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "instance": "10.142.0.43:9100" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "12303.4" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "instance": "10.142.0.107:9100" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "instance": "10.142.0.43:9100" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "13011002368" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG { 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "ds-idrepo-1" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "ds-idrepo-0" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "ds-idrepo-1" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "ds-idrepo-0" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG }, 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG "value": [ 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "413621" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0.037" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "instance": "10.142.0.107:9100" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "instance": "10.142.0.43:9100" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0.000333333333333267" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "2283.8333333333335" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "2283.9333333333334" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "ds-idrepo-2" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "ds-idrepo-1" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "ds-idrepo-0" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "1733738496" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095923" 12:58:43 INFO [http_cmd]: http status code OK 12:58:43 DEBUG --- status code --- 12:58:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:43 DEBUG --- http response --- 12:58:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:43 DEBUG ----- output ----- 12:58:43 DEBUG { 12:58:43 DEBUG "status": "success", 12:58:43 DEBUG "data": { 12:58:43 DEBUG "resultType": "vector", 12:58:43 DEBUG "result": [ 12:58:43 DEBUG { 12:58:43 DEBUG "metric": { 12:58:43 DEBUG "pod": "ds-idrepo-1" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG [...] 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG 12:58:43 DEBUG "pod": "ds-idrepo-0" 12:58:43 DEBUG }, 12:58:43 DEBUG "value": [ 12:58:43 DEBUG 1713095923, 12:58:43 DEBUG "0" 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG ] 12:58:43 DEBUG } 12:58:43 DEBUG } 12:58:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095924" 12:58:44 INFO [http_cmd]: http status code OK 12:58:44 DEBUG --- status code --- 12:58:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:44 DEBUG --- http response --- 12:58:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:44 DEBUG ----- output ----- 12:58:44 DEBUG { 12:58:44 DEBUG "status": "success", 12:58:44 DEBUG "data": { 12:58:44 DEBUG "resultType": "vector", 12:58:44 DEBUG "result": [ 12:58:44 DEBUG { 12:58:44 DEBUG "metric": { 12:58:44 DEBUG "instance": "10.142.0.107:9100" 12:58:44 DEBUG }, 12:58:44 DEBUG "value": [ 12:58:44 DEBUG 12:58:44 DEBUG 12:58:44 DEBUG 12:58:44 DEBUG [...] 12:58:44 DEBUG 12:58:44 DEBUG 12:58:44 DEBUG 12:58:44 DEBUG "instance": "10.142.0.43:9100" 12:58:44 DEBUG }, 12:58:44 DEBUG "value": [ 12:58:44 DEBUG 1713095924, 12:58:44 DEBUG "0" 12:58:44 DEBUG ] 12:58:44 DEBUG } 12:58:44 DEBUG ] 12:58:44 DEBUG } 12:58:44 DEBUG } 12:58:59 INFO 12:58:59 INFO [loop_until]: kubectl --namespace=xlou top pods 12:58:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:58:59 INFO [loop_until]: OK (rc = 0) 12:58:59 DEBUG --- stdout --- 12:58:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12916m 8623Mi am-7d597cf7d6-8vr8f 11898m 8652Mi am-7d597cf7d6-9wz9h 15444m 8986Mi ds-cts-0 11474m 8146Mi ds-cts-1 12450m 8393Mi ds-cts-2 12141m 8162Mi ds-idrepo-0 13400m 4555Mi ds-idrepo-1 1972m 4318Mi ds-idrepo-2 1928m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6526m 1562Mi 12:58:59 DEBUG --- stderr --- 12:58:59 DEBUG 12:59:06 INFO 12:59:06 INFO [loop_until]: kubectl --namespace=xlou top node 12:59:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:59:06 INFO [loop_until]: OK (rc = 0) 12:59:06 DEBUG --- stdout --- 12:59:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4882Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11953m 40% 10799Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13170m 44% 10691Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17627m 59% 3052Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2094m 7% 6603Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11787m 39% 12027Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2929Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12481m 41% 10112Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2015m 6% 5734Mi 5% gke-xlou-cdm-ds-0f601751-j488 12239m 40% 10362Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 68m 0% 2213Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16864m 56% 10564Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13462m 45% 6232Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6858m 43% 5153Mi 8% 12:59:06 DEBUG --- stderr --- 12:59:06 DEBUG 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "2289.3" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "6.687163441404614" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "ds-idrepo-1" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "ds-cts-1" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "0.000033333333333333335" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "instance": "10.142.0.107:9100" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "instance": "10.142.0.43:9100" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "6745.966666666666" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "ds-idrepo-1" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "ds-cts-1" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "0" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 ERROR [http_cmd]: ERROR 12:59:43 ERROR --- status code --- 12:59:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:59:43 ERROR --- http response --- 12:59:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:59:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "0" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "instance": "10.142.0.107:9100" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "instance": "10.142.0.43:9100" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "12363.966666666664" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "2305.0666666666666" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "ds-idrepo-1" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "ds-idrepo-0" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "0" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "instance": "10.142.0.107:9100" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "instance": "10.142.0.43:9100" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "13011547136" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "ds-idrepo-1" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "4807.511737089202" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG { 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "data": { 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG "result": [ 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG { 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:59:43 DEBUG "pod": "ds-idrepo-1" 12:59:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:59:43 DEBUG }, 12:59:43 DEBUG }, 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG "value": [ 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG [...] 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "ds-idrepo-0" 12:59:43 DEBUG "pod": "ds-idrepo-0" 12:59:43 DEBUG "pod": "ds-idrepo-0" 12:59:43 DEBUG }, 12:59:43 DEBUG }, 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG "value": [ 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "0" 12:59:43 DEBUG "0" 12:59:43 DEBUG "0" 12:59:43 DEBUG ] 12:59:43 DEBUG ] 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG ] 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "ds-idrepo-1" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "ds-idrepo-0" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "0" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG { 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "413080.5" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "0.037" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:59:43 DEBUG "instance": "10.142.0.107:9100" 12:59:43 DEBUG }, 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:59:43 DEBUG "instance": "10.142.0.43:9100" 12:59:43 DEBUG }, 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG 1713095983, 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 DEBUG "2290.9666666666667" 12:59:43 DEBUG "0.00033333333333350387" 12:59:43 DEBUG ] 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG ] 12:59:43 DEBUG ] 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "1731792896" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "2285.6" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "ds-idrepo-1" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "0" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "ds-idrepo-1" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "ds-idrepo-0" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "0" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713095983" 12:59:43 INFO [http_cmd]: http status code OK 12:59:43 DEBUG --- status code --- 12:59:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:43 DEBUG --- http response --- 12:59:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:43 DEBUG ----- output ----- 12:59:43 DEBUG { 12:59:43 DEBUG "status": "success", 12:59:43 DEBUG "data": { 12:59:43 DEBUG "resultType": "vector", 12:59:43 DEBUG "result": [ 12:59:43 DEBUG { 12:59:43 DEBUG "metric": { 12:59:43 DEBUG "pod": "ds-idrepo-1" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG [...] 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG 12:59:43 DEBUG "pod": "ds-idrepo-0" 12:59:43 DEBUG }, 12:59:43 DEBUG "value": [ 12:59:43 DEBUG 1713095983, 12:59:43 DEBUG "0" 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG ] 12:59:43 DEBUG } 12:59:43 DEBUG } 12:59:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713095984" 12:59:44 INFO [http_cmd]: http status code OK 12:59:44 DEBUG --- status code --- 12:59:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:44 DEBUG --- http response --- 12:59:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:44 DEBUG ----- output ----- 12:59:44 DEBUG { 12:59:44 DEBUG "status": "success", 12:59:44 DEBUG "data": { 12:59:44 DEBUG "resultType": "vector", 12:59:44 DEBUG "result": [ 12:59:44 DEBUG { 12:59:44 DEBUG "metric": { 12:59:44 DEBUG "instance": "10.142.0.107:9100" 12:59:44 DEBUG }, 12:59:44 DEBUG "value": [ 12:59:44 DEBUG 12:59:44 DEBUG 12:59:44 DEBUG 12:59:44 DEBUG [...] 12:59:44 DEBUG 12:59:44 DEBUG 12:59:44 DEBUG 12:59:44 DEBUG "instance": "10.142.0.43:9100" 12:59:44 DEBUG }, 12:59:44 DEBUG "value": [ 12:59:44 DEBUG 1713095984, 12:59:44 DEBUG "0" 12:59:44 DEBUG ] 12:59:44 DEBUG } 12:59:44 DEBUG ] 12:59:44 DEBUG } 12:59:44 DEBUG } 13:00:00 INFO 13:00:00 INFO [loop_until]: kubectl --namespace=xlou top pods 13:00:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:00:00 INFO [loop_until]: OK (rc = 0) 13:00:00 DEBUG --- stdout --- 13:00:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12403m 8623Mi am-7d597cf7d6-8vr8f 11618m 8654Mi am-7d597cf7d6-9wz9h 15892m 9020Mi ds-cts-0 11639m 8129Mi ds-cts-1 11850m 8394Mi ds-cts-2 11832m 8190Mi ds-idrepo-0 13259m 4555Mi ds-idrepo-1 2017m 4318Mi ds-idrepo-2 1898m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6510m 1561Mi 13:00:00 DEBUG --- stderr --- 13:00:00 DEBUG 13:00:06 INFO 13:00:06 INFO [loop_until]: kubectl --namespace=xlou top node 13:00:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:00:06 INFO [loop_until]: OK (rc = 0) 13:00:06 DEBUG --- stdout --- 13:00:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4887Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12204m 40% 10803Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12949m 43% 10693Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17625m 59% 3051Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2128m 7% 6596Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11362m 38% 12045Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2932Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12334m 41% 10124Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2010m 6% 5742Mi 5% gke-xlou-cdm-ds-0f601751-j488 12339m 41% 10355Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2226Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 17037m 57% 10595Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13365m 44% 6237Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6840m 43% 5151Mi 8% 13:00:06 DEBUG --- stderr --- 13:00:06 DEBUG 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:00:43 DEBUG "pod": "ds-idrepo-1" 13:00:43 DEBUG }, 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG [...] 13:00:43 DEBUG [...] 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG 13:00:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:00:43 DEBUG { 13:00:43 DEBUG "pod": "ds-cts-1" 13:00:43 DEBUG }, 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG "data": { 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "2280.9333333333334" 13:00:43 DEBUG "result": [ 13:00:43 DEBUG "0.000018208333333333333" 13:00:43 DEBUG ] 13:00:43 DEBUG { 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG "metric": { 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG }, 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 DEBUG "value": [ 13:00:43 DEBUG } 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "6.709236122692074" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "0" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "instance": "10.142.0.107:9100" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "instance": "10.142.0.43:9100" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "6886.599999999999" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 ERROR [http_cmd]: ERROR 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 ERROR --- status code --- 13:00:43 DEBUG --- status code --- 13:00:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 ERROR --- http response --- 13:00:43 DEBUG --- http response --- 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:43 DEBUG --- status code --- 13:00:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG { 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "2296.7666666666664" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG { 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "instance": "10.142.0.107:9100" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "instance": "10.142.0.43:9100" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "12404.299999999997" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "ds-idrepo-1" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "ds-cts-1" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "0" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "ds-idrepo-1" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "ds-idrepo-0" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "0" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "instance": "10.142.0.107:9100" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "instance": "10.142.0.43:9100" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "13012339712" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "ds-idrepo-0" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "3937.6391171757896" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "ds-idrepo-1" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "ds-idrepo-0" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "0" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "ds-idrepo-0" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "9.348595844668086" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "ds-idrepo-0" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "6075.214637928361" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "412771.5" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "1733214208" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "ds-idrepo-1" 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:43 DEBUG }, 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG "value": [ 13:00:43 DEBUG { 13:00:43 DEBUG 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG 13:00:43 DEBUG "data": { 13:00:43 DEBUG 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG [...] 13:00:43 DEBUG "result": [ 13:00:43 DEBUG 13:00:43 DEBUG { 13:00:43 DEBUG 13:00:43 DEBUG "metric": { 13:00:43 DEBUG 13:00:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:00:43 DEBUG "pod": "ds-idrepo-0" 13:00:43 DEBUG }, 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG 13:00:43 DEBUG "0" 13:00:43 DEBUG 13:00:43 DEBUG ] 13:00:43 DEBUG [...] 13:00:43 DEBUG } 13:00:43 DEBUG 13:00:43 DEBUG ] 13:00:43 DEBUG 13:00:43 DEBUG } 13:00:43 DEBUG 13:00:43 DEBUG } 13:00:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "0.037" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "2280.366666666667" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG { 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "ds-idrepo-0" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "0" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "instance": "10.142.0.107:9100" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "instance": "10.142.0.43:9100" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "0.000333333333333267" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "2280.4666666666667" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "ds-idrepo-1" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "ds-idrepo-0" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "0" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096043" 13:00:43 INFO [http_cmd]: http status code OK 13:00:43 DEBUG --- status code --- 13:00:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:43 DEBUG --- http response --- 13:00:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:43 DEBUG ----- output ----- 13:00:43 DEBUG { 13:00:43 DEBUG "status": "success", 13:00:43 DEBUG "data": { 13:00:43 DEBUG "resultType": "vector", 13:00:43 DEBUG "result": [ 13:00:43 DEBUG { 13:00:43 DEBUG "metric": { 13:00:43 DEBUG "pod": "ds-idrepo-1" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG [...] 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG 13:00:43 DEBUG "pod": "ds-idrepo-0" 13:00:43 DEBUG }, 13:00:43 DEBUG "value": [ 13:00:43 DEBUG 1713096043, 13:00:43 DEBUG "0" 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG ] 13:00:43 DEBUG } 13:00:43 DEBUG } 13:00:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096044" 13:00:44 INFO [http_cmd]: http status code OK 13:00:44 DEBUG --- status code --- 13:00:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:44 DEBUG --- http response --- 13:00:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:44 DEBUG ----- output ----- 13:00:44 DEBUG { 13:00:44 DEBUG "status": "success", 13:00:44 DEBUG "data": { 13:00:44 DEBUG "resultType": "vector", 13:00:44 DEBUG "result": [ 13:00:44 DEBUG { 13:00:44 DEBUG "metric": { 13:00:44 DEBUG "instance": "10.142.0.107:9100" 13:00:44 DEBUG }, 13:00:44 DEBUG "value": [ 13:00:44 DEBUG 13:00:44 DEBUG 13:00:44 DEBUG 13:00:44 DEBUG [...] 13:00:44 DEBUG 13:00:44 DEBUG 13:00:44 DEBUG 13:00:44 DEBUG "instance": "10.142.0.43:9100" 13:00:44 DEBUG }, 13:00:44 DEBUG "value": [ 13:00:44 DEBUG 1713096044, 13:00:44 DEBUG "0" 13:00:44 DEBUG ] 13:00:44 DEBUG } 13:00:44 DEBUG ] 13:00:44 DEBUG } 13:00:44 DEBUG } 13:01:00 INFO 13:01:00 INFO [loop_until]: kubectl --namespace=xlou top pods 13:01:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:01:00 INFO [loop_until]: OK (rc = 0) 13:01:00 DEBUG --- stdout --- 13:01:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12424m 8624Mi am-7d597cf7d6-8vr8f 11301m 8653Mi am-7d597cf7d6-9wz9h 15982m 8992Mi ds-cts-0 11518m 8127Mi ds-cts-1 12205m 8393Mi ds-cts-2 12125m 8165Mi ds-idrepo-0 12992m 4555Mi ds-idrepo-1 1998m 4318Mi ds-idrepo-2 1829m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6869m 1562Mi 13:01:00 DEBUG --- stderr --- 13:01:00 DEBUG 13:01:06 INFO 13:01:06 INFO [loop_until]: kubectl --namespace=xlou top node 13:01:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:01:06 INFO [loop_until]: OK (rc = 0) 13:01:06 DEBUG --- stdout --- 13:01:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4885Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11802m 39% 10806Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12920m 43% 10690Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17010m 56% 3032Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2070m 6% 6602Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11596m 38% 12019Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2928Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12343m 41% 10126Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1980m 6% 5737Mi 5% gke-xlou-cdm-ds-0f601751-j488 12093m 40% 10367Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2224Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16918m 56% 10564Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13373m 44% 6238Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6604m 41% 5152Mi 8% 13:01:06 DEBUG --- stderr --- 13:01:06 DEBUG 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG { 13:01:43 DEBUG { 13:01:43 DEBUG { 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG "data": { 13:01:43 DEBUG "data": { 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG "result": [ 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG { 13:01:43 DEBUG { 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:01:43 DEBUG "data": { 13:01:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:01:43 DEBUG "pod": "ds-idrepo-1" 13:01:43 DEBUG }, 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG }, 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG "result": [ 13:01:43 DEBUG "value": [ 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG { 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "metric": { 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG }, 13:01:43 DEBUG [...] 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:01:43 DEBUG 13:01:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:01:43 DEBUG "pod": "ds-cts-1" 13:01:43 DEBUG }, 13:01:43 DEBUG [...] 13:01:43 DEBUG }, 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG "value": [ 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "2311.3333333333335" 13:01:43 DEBUG 13:01:43 DEBUG "0" 13:01:43 DEBUG "0.0003666666666666666" 13:01:43 DEBUG ] 13:01:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:01:43 DEBUG ] 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG }, 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG "value": [ 13:01:43 DEBUG ] 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 DEBUG "6.703730352232557" 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "instance": "10.142.0.107:9100" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "instance": "10.142.0.43:9100" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "5254.766666666666" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 ERROR [http_cmd]: ERROR 13:01:43 ERROR --- status code --- 13:01:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:01:43 ERROR --- http response --- 13:01:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:01:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "ds-idrepo-1" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "ds-cts-1" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "0" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "ds-idrepo-1" 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:01:43 DEBUG }, 13:01:43 DEBUG "instance": "10.142.0.107:9100" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "ds-idrepo-0" 13:01:43 DEBUG 13:01:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:01:43 DEBUG }, 13:01:43 DEBUG "instance": "10.142.0.43:9100" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "0" 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "2284.4666666666667" 13:01:43 DEBUG ] 13:01:43 DEBUG "12293.333333333332" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "instance": "10.142.0.107:9100" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "instance": "10.142.0.43:9100" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "13013068800" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "ds-idrepo-1" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "ds-idrepo-0" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "0" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "0" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "42923037.75737301" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "13.330793208461316" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "413068.5" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "0.037" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "1734475776" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "2308.9666666666667" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "instance": "10.142.0.107:9100" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "instance": "10.142.0.43:9100" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "0.000500000000000019" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "ds-idrepo-1" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "ds-idrepo-0" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "0" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "2310.4" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "0" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "ds-idrepo-1" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "ds-idrepo-0" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "0" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096103" 13:01:43 INFO [http_cmd]: http status code OK 13:01:43 DEBUG --- status code --- 13:01:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:43 DEBUG --- http response --- 13:01:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:43 DEBUG ----- output ----- 13:01:43 DEBUG { 13:01:43 DEBUG "status": "success", 13:01:43 DEBUG "data": { 13:01:43 DEBUG "resultType": "vector", 13:01:43 DEBUG "result": [ 13:01:43 DEBUG { 13:01:43 DEBUG "metric": { 13:01:43 DEBUG "pod": "ds-idrepo-1" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG [...] 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG 13:01:43 DEBUG "pod": "ds-idrepo-0" 13:01:43 DEBUG }, 13:01:43 DEBUG "value": [ 13:01:43 DEBUG 1713096103, 13:01:43 DEBUG "0" 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG ] 13:01:43 DEBUG } 13:01:43 DEBUG } 13:01:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096104" 13:01:44 INFO [http_cmd]: http status code OK 13:01:44 DEBUG --- status code --- 13:01:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:44 DEBUG --- http response --- 13:01:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:44 DEBUG ----- output ----- 13:01:44 DEBUG { 13:01:44 DEBUG "status": "success", 13:01:44 DEBUG "data": { 13:01:44 DEBUG "resultType": "vector", 13:01:44 DEBUG "result": [ 13:01:44 DEBUG { 13:01:44 DEBUG "metric": { 13:01:44 DEBUG "instance": "10.142.0.107:9100" 13:01:44 DEBUG }, 13:01:44 DEBUG "value": [ 13:01:44 DEBUG 13:01:44 DEBUG 13:01:44 DEBUG 13:01:44 DEBUG [...] 13:01:44 DEBUG 13:01:44 DEBUG 13:01:44 DEBUG 13:01:44 DEBUG "instance": "10.142.0.43:9100" 13:01:44 DEBUG }, 13:01:44 DEBUG "value": [ 13:01:44 DEBUG 1713096104, 13:01:44 DEBUG "0.00033333333333333365" 13:01:44 DEBUG ] 13:01:44 DEBUG } 13:01:44 DEBUG ] 13:01:44 DEBUG } 13:01:44 DEBUG } 13:02:00 INFO 13:02:00 INFO [loop_until]: kubectl --namespace=xlou top pods 13:02:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:02:00 INFO [loop_until]: OK (rc = 0) 13:02:00 DEBUG --- stdout --- 13:02:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13137m 8674Mi am-7d597cf7d6-8vr8f 11324m 8650Mi am-7d597cf7d6-9wz9h 14990m 8983Mi ds-cts-0 11924m 8122Mi ds-cts-1 10958m 8400Mi ds-cts-2 12143m 8153Mi ds-idrepo-0 13453m 4555Mi ds-idrepo-1 1791m 4318Mi ds-idrepo-2 1907m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6553m 1561Mi 13:02:00 DEBUG --- stderr --- 13:02:00 DEBUG 13:02:06 INFO 13:02:06 INFO [loop_until]: kubectl --namespace=xlou top node 13:02:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:02:06 INFO [loop_until]: OK (rc = 0) 13:02:06 DEBUG --- stdout --- 13:02:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4880Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12220m 40% 10771Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11607m 38% 10746Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16588m 55% 3056Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2091m 7% 6602Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 10988m 36% 12015Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2914Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 11061m 37% 10110Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1910m 6% 5747Mi 5% gke-xlou-cdm-ds-0f601751-j488 11485m 38% 10365Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2222Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 17024m 57% 10578Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13394m 44% 6231Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6844m 43% 5156Mi 8% 13:02:06 DEBUG --- stderr --- 13:02:06 DEBUG 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096163" 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "274.93333333333334" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096163" 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "ds-idrepo-1" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "ds-cts-1" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "0.00013795833333333333" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG { 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "0" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "6.689397415990259" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "instance": "10.142.0.107:9100" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "instance": "10.142.0.43:9100" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "5352.733333333333" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "ds-idrepo-1" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "ds-idrepo-0" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "0" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096163" 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096163" 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 ERROR [http_cmd]: ERROR 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 ERROR --- status code --- 13:02:43 DEBUG --- status code --- 13:02:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 ERROR --- http response --- 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:02:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "ds-idrepo-1" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "ds-cts-1" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "0" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "instance": "10.142.0.107:9100" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "instance": "10.142.0.43:9100" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "11900.633333333331" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "ds-idrepo-1" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "ds-idrepo-0" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "0" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "2288.4333333333334" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096163" 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "instance": "10.142.0.107:9100" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "instance": "10.142.0.43:9100" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "13013607424" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "4252497.6243357975" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "254262001.1808035" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "4.292059490005111" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "383548.5" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "0.037" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096163" 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096163" 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "1734041600" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "273.96666666666664" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096163" 13:02:43 INFO [http_cmd]: http status code OK 13:02:43 DEBUG --- status code --- 13:02:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:43 DEBUG --- http response --- 13:02:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:43 DEBUG ----- output ----- 13:02:43 DEBUG { 13:02:43 DEBUG "status": "success", 13:02:43 DEBUG "data": { 13:02:43 DEBUG "resultType": "vector", 13:02:43 DEBUG "result": [ 13:02:43 DEBUG { 13:02:43 DEBUG "metric": { 13:02:43 DEBUG "instance": "10.142.0.107:9100" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG [...] 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG 13:02:43 DEBUG "instance": "10.142.0.43:9100" 13:02:43 DEBUG }, 13:02:43 DEBUG "value": [ 13:02:43 DEBUG 1713096163, 13:02:43 DEBUG "0.00030000000000001136" 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG ] 13:02:43 DEBUG } 13:02:43 DEBUG } 13:02:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096164" 13:02:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096164" 13:02:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096164" 13:02:44 INFO [http_cmd]: http status code OK 13:02:44 DEBUG --- status code --- 13:02:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:44 DEBUG --- http response --- 13:02:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:44 DEBUG ----- output ----- 13:02:44 DEBUG { 13:02:44 DEBUG "status": "success", 13:02:44 DEBUG "data": { 13:02:44 DEBUG "resultType": "vector", 13:02:44 DEBUG "result": [ 13:02:44 DEBUG { 13:02:44 DEBUG "metric": { 13:02:44 DEBUG "pod": "ds-idrepo-1" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG [...] 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG "pod": "ds-idrepo-0" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 1713096164, 13:02:44 DEBUG "0" 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG } 13:02:44 INFO [http_cmd]: http status code OK 13:02:44 DEBUG --- status code --- 13:02:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:44 DEBUG --- http response --- 13:02:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:44 DEBUG ----- output ----- 13:02:44 DEBUG { 13:02:44 DEBUG "status": "success", 13:02:44 DEBUG "data": { 13:02:44 DEBUG "resultType": "vector", 13:02:44 DEBUG "result": [ 13:02:44 DEBUG { 13:02:44 DEBUG "metric": { 13:02:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG [...] 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 1713096164, 13:02:44 DEBUG "274.2" 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG } 13:02:44 INFO [http_cmd]: http status code OK 13:02:44 DEBUG --- status code --- 13:02:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:44 DEBUG --- http response --- 13:02:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:02:44 DEBUG ----- output ----- 13:02:44 DEBUG { 13:02:44 DEBUG "status": "success", 13:02:44 DEBUG "data": { 13:02:44 DEBUG "resultType": "vector", 13:02:44 DEBUG "result": [ 13:02:44 DEBUG { 13:02:44 DEBUG "metric": { 13:02:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG [...] 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 1713096164, 13:02:44 DEBUG "0" 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG } 13:02:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096164" 13:02:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096164" 13:02:44 INFO [http_cmd]: http status code OK 13:02:44 DEBUG --- status code --- 13:02:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:44 DEBUG --- http response --- 13:02:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:44 DEBUG ----- output ----- 13:02:44 DEBUG { 13:02:44 DEBUG "status": "success", 13:02:44 DEBUG "data": { 13:02:44 DEBUG "resultType": "vector", 13:02:44 DEBUG "result": [ 13:02:44 DEBUG { 13:02:44 DEBUG "metric": { 13:02:44 DEBUG "pod": "ds-idrepo-1" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG [...] 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG "pod": "ds-idrepo-0" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 1713096164, 13:02:44 DEBUG "0" 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG } 13:02:44 INFO [http_cmd]: http status code OK 13:02:44 DEBUG --- status code --- 13:02:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:44 DEBUG --- http response --- 13:02:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:44 DEBUG ----- output ----- 13:02:44 DEBUG { 13:02:44 DEBUG "status": "success", 13:02:44 DEBUG "data": { 13:02:44 DEBUG "resultType": "vector", 13:02:44 DEBUG "result": [ 13:02:44 DEBUG { 13:02:44 DEBUG "metric": { 13:02:44 DEBUG "pod": "ds-idrepo-1" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG [...] 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG "pod": "ds-idrepo-0" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 1713096164, 13:02:44 DEBUG "0" 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG } 13:02:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096164" 13:02:44 INFO [http_cmd]: http status code OK 13:02:44 DEBUG --- status code --- 13:02:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:44 DEBUG --- http response --- 13:02:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:44 DEBUG ----- output ----- 13:02:44 DEBUG { 13:02:44 DEBUG "status": "success", 13:02:44 DEBUG "data": { 13:02:44 DEBUG "resultType": "vector", 13:02:44 DEBUG "result": [ 13:02:44 DEBUG { 13:02:44 DEBUG "metric": { 13:02:44 DEBUG "instance": "10.142.0.107:9100" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG [...] 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG 13:02:44 DEBUG "instance": "10.142.0.43:9100" 13:02:44 DEBUG }, 13:02:44 DEBUG "value": [ 13:02:44 DEBUG 1713096164, 13:02:44 DEBUG "0" 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG ] 13:02:44 DEBUG } 13:02:44 DEBUG } 13:03:00 INFO 13:03:00 INFO [loop_until]: kubectl --namespace=xlou top pods 13:03:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:03:00 INFO [loop_until]: OK (rc = 0) 13:03:00 DEBUG --- stdout --- 13:03:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 2149m 8549Mi am-7d597cf7d6-8vr8f 2019m 8652Mi am-7d597cf7d6-9wz9h 2697m 8997Mi ds-cts-0 5350m 8115Mi ds-cts-1 5585m 8411Mi ds-cts-2 5798m 8173Mi ds-idrepo-0 1245m 4555Mi ds-idrepo-1 172m 4317Mi ds-idrepo-2 167m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-58bdc8d6d4-hh8r9 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 585Mi 13:03:00 DEBUG --- stderr --- 13:03:00 DEBUG 13:03:06 INFO 13:03:06 INFO [loop_until]: kubectl --namespace=xlou top node 13:03:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:03:06 INFO [loop_until]: OK (rc = 0) 13:03:06 DEBUG --- stdout --- 13:03:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 4881Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5813m 19% 10783Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2202m 7% 10682Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 91m 0% 3043Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 211m 0% 6594Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5282m 17% 12029Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2922Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 2189m 7% 10109Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 227m 0% 5742Mi 5% gke-xlou-cdm-ds-0f601751-j488 5746m 19% 10403Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2218Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 2915m 9% 10648Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1323m 4% 6233Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 66m 0% 4187Mi 7% 13:03:06 DEBUG --- stderr --- 13:03:06 DEBUG 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096223" 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096223" 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG { 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:43 DEBUG { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG "data": { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG "data": { 13:03:43 DEBUG "result": [ 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG { 13:03:43 DEBUG "result": [ 13:03:43 DEBUG "metric": { 13:03:43 DEBUG { 13:03:43 DEBUG "pod": "ds-idrepo-1" 13:03:43 DEBUG "metric": { 13:03:43 DEBUG }, 13:03:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:03:43 DEBUG "value": [ 13:03:43 DEBUG }, 13:03:43 DEBUG 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG [...] 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG [...] 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG "pod": "ds-cts-1" 13:03:43 DEBUG 13:03:43 DEBUG }, 13:03:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:03:43 DEBUG "value": [ 13:03:43 DEBUG }, 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG "0.00006975" 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG ] 13:03:43 DEBUG "0" 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG "data": { 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG "result": [ 13:03:43 DEBUG { 13:03:43 DEBUG "metric": { 13:03:43 DEBUG "pod": "ds-idrepo-1" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG [...] 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG "pod": "ds-idrepo-0" 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG }, 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG "value": [ 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG "0" 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG ] 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG } 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG ] 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG { 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG } 13:03:43 DEBUG { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG } 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG "data": { 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:43 DEBUG "data": { 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG "result": [ 13:03:43 DEBUG { 13:03:43 DEBUG "result": [ 13:03:43 DEBUG { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG { 13:03:43 DEBUG "metric": { 13:03:43 DEBUG "data": { 13:03:43 DEBUG "metric": { 13:03:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG "instance": "10.142.0.107:9100" 13:03:43 DEBUG }, 13:03:43 DEBUG "result": [ 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG { 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 13:03:43 DEBUG "metric": { 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG }, 13:03:43 DEBUG 13:03:43 DEBUG [...] 13:03:43 DEBUG "value": [ 13:03:43 DEBUG [...] 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:03:43 DEBUG [...] 13:03:43 DEBUG "instance": "10.142.0.43:9100" 13:03:43 DEBUG }, 13:03:43 DEBUG 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG "0" 13:03:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:03:43 DEBUG "5263.7" 13:03:43 DEBUG ] 13:03:43 DEBUG }, 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG "value": [ 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG "0.02702351119039655" 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096223" 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096223" 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096223" 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG "data": { 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG "result": [ 13:03:43 DEBUG { 13:03:43 DEBUG "metric": { 13:03:43 DEBUG "pod": "ds-idrepo-1" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG [...] 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG "pod": "ds-cts-1" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG "0" 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG "data": { 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG "result": [ 13:03:43 DEBUG { 13:03:43 DEBUG "metric": { 13:03:43 DEBUG "pod": "ds-idrepo-1" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG [...] 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG "pod": "ds-idrepo-0" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG "0" 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG "data": { 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG "result": [ 13:03:43 DEBUG { 13:03:43 DEBUG "metric": { 13:03:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG [...] 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG "2281.8333333333335" 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 ERROR [http_cmd]: ERROR 13:03:43 ERROR --- status code --- 13:03:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:03:43 ERROR --- http response --- 13:03:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:03:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG "data": { 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG "result": [ 13:03:43 DEBUG { 13:03:43 DEBUG "metric": { 13:03:43 DEBUG "instance": "10.142.0.107:9100" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG [...] 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG "instance": "10.142.0.43:9100" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG "11896.533333333331" 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG "data": { 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG "result": [ 13:03:43 DEBUG { 13:03:43 DEBUG "metric": { 13:03:43 DEBUG "instance": "10.142.0.107:9100" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG [...] 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG "instance": "10.142.0.43:9100" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG "13014289408" 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:43 INFO [http_cmd]: http status code OK 13:03:43 DEBUG --- status code --- 13:03:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:43 DEBUG --- http response --- 13:03:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:03:43 DEBUG ----- output ----- 13:03:43 DEBUG { 13:03:43 DEBUG "status": "success", 13:03:43 DEBUG "data": { 13:03:43 DEBUG "resultType": "vector", 13:03:43 DEBUG "result": [ 13:03:43 DEBUG { 13:03:43 DEBUG "metric": { 13:03:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG [...] 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG 13:03:43 DEBUG "pod": "idm-b85f9d978-rl76f" 13:03:43 DEBUG }, 13:03:43 DEBUG "value": [ 13:03:43 DEBUG 1713096223, 13:03:43 DEBUG "585.9171047455566" 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG ] 13:03:43 DEBUG } 13:03:43 DEBUG } 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096223" 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "metric": { 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096224" 13:03:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:03:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:03:44 DEBUG }, 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG [...] 13:03:44 DEBUG [...] 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "pod": "idm-b85f9d978-rl76f" 13:03:44 DEBUG "pod": "idm-b85f9d978-rl76f" 13:03:44 DEBUG }, 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096223, 13:03:44 DEBUG 1713096223, 13:03:44 DEBUG "13915.531237706968" 13:03:44 DEBUG "0.0029476451024568404" 13:03:44 DEBUG ] 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "instance": "10.142.0.107:9100" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG [...] 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "instance": "10.142.0.43:9100" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "0.0004333333333332708" 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096224" 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096224" 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG [...] 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "250043.5" 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG [...] 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "0.037" 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096224" 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096224" 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096224" 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG [...] 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "712114176" 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG [...] 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "0" 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "pod": "ds-idrepo-1" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG [...] 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "pod": "ds-idrepo-0" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "0" 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096224" 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096224" 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096224" 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "pod": "ds-idrepo-1" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG [...] 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "pod": "ds-idrepo-0" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "0" 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:03:44 DEBUG 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG 13:03:44 DEBUG { 13:03:44 DEBUG 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG [...] 13:03:44 DEBUG "data": { 13:03:44 DEBUG 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG 13:03:44 DEBUG "result": [ 13:03:44 DEBUG 13:03:44 DEBUG { 13:03:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:03:44 DEBUG "metric": { 13:03:44 DEBUG }, 13:03:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:03:44 DEBUG "value": [ 13:03:44 DEBUG }, 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG "0" 13:03:44 DEBUG 13:03:44 DEBUG ] 13:03:44 DEBUG 13:03:44 DEBUG } 13:03:44 DEBUG 13:03:44 DEBUG ] 13:03:44 DEBUG [...] 13:03:44 DEBUG } 13:03:44 DEBUG 13:03:44 DEBUG } 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "pod": "idm-b85f9d978-rl76f" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "0" 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096224" 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "pod": "ds-idrepo-1" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG [...] 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "pod": "ds-idrepo-0" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "0" 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096224" 13:03:44 INFO [http_cmd]: http status code OK 13:03:44 DEBUG --- status code --- 13:03:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:44 DEBUG --- http response --- 13:03:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:44 DEBUG ----- output ----- 13:03:44 DEBUG { 13:03:44 DEBUG "status": "success", 13:03:44 DEBUG "data": { 13:03:44 DEBUG "resultType": "vector", 13:03:44 DEBUG "result": [ 13:03:44 DEBUG { 13:03:44 DEBUG "metric": { 13:03:44 DEBUG "instance": "10.142.0.107:9100" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG [...] 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG 13:03:44 DEBUG "instance": "10.142.0.43:9100" 13:03:44 DEBUG }, 13:03:44 DEBUG "value": [ 13:03:44 DEBUG 1713096224, 13:03:44 DEBUG "0.0006666666666666673" 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG ] 13:03:44 DEBUG } 13:03:44 DEBUG } 13:03:56 WARNING File /tmp/lodemon_data-20240414_063908/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [14/Apr/2024 13:03:56] "GET /monitoring/average?start_time=20240414_110311&stop_time=20240414_120155 HTTP/1.1" 200 - 13:04:00 INFO 13:04:00 INFO [loop_until]: kubectl --namespace=xlou top pods 13:04:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:04:00 INFO [loop_until]: OK (rc = 0) 13:04:00 DEBUG --- stdout --- 13:04:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 2093m 8544Mi am-7d597cf7d6-8vr8f 2010m 8618Mi am-7d597cf7d6-9wz9h 2569m 8933Mi ds-cts-0 5049m 8100Mi ds-cts-1 5779m 8424Mi ds-cts-2 5446m 8158Mi ds-idrepo-0 1300m 4555Mi ds-idrepo-1 170m 4318Mi ds-idrepo-2 160m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 6m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 3m 585Mi 13:04:00 DEBUG --- stderr --- 13:04:00 DEBUG 13:04:06 INFO 13:04:06 INFO [loop_until]: kubectl --namespace=xlou top node 13:04:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:04:06 INFO [loop_until]: OK (rc = 0) 13:04:06 DEBUG --- stdout --- 13:04:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4885Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5467m 18% 10799Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2204m 7% 10634Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3030Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 215m 0% 6594Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5212m 17% 12016Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2924Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 2228m 7% 10102Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 223m 0% 5748Mi 5% gke-xlou-cdm-ds-0f601751-j488 5427m 18% 10406Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2223Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 2875m 9% 10556Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1359m 4% 6244Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 84m 0% 4187Mi 7% 13:04:06 DEBUG --- stderr --- 13:04:06 DEBUG 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096283" 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096283" 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096283" 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096283" 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096283" 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG { 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "data": { 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "result": [ 13:04:43 DEBUG { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "pod": "ds-idrepo-1" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG "pod": "ds-idrepo-0" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG "0" 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG { 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "data": { 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "result": [ 13:04:43 DEBUG { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 1713096283, 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG "0.027860620937779544" 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG ] 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG } 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG ] 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG } 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG } 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG { 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "data": { 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "result": [ 13:04:43 DEBUG { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "pod": "ds-idrepo-1" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG "pod": "ds-cts-1" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG "0.000018208333333333333" 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG { 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "data": { 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "result": [ 13:04:43 DEBUG { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "instance": "10.142.0.107:9100" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG "instance": "10.142.0.43:9100" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG "14483.566666666668" 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG { 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "data": { 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "result": [ 13:04:43 DEBUG { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG "0" 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096283" 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG { 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "data": { 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "result": [ 13:04:43 DEBUG { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG "0" 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096283" 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG { 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "data": { 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "result": [ 13:04:43 DEBUG { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG "2311.4" 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096283" 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096283" 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG --- http response --- 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096283" 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG { 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "data": { 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "result": [ 13:04:43 DEBUG { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG "0" 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096283" 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096283" 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096283" 13:04:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096283" 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG { 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "data": { 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "result": [ 13:04:43 DEBUG { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "pod": "ds-idrepo-1" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG "pod": "ds-cts-1" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG "0" 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 ERROR [http_cmd]: ERROR 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG --- status code --- 13:04:43 ERROR --- status code --- 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG --- http response --- 13:04:43 ERROR --- http response --- 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:04:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:43 DEBUG ----- output ----- 13:04:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG { 13:04:43 DEBUG { 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "data": { 13:04:43 DEBUG "data": { 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG "result": [ 13:04:43 DEBUG "result": [ 13:04:43 DEBUG { 13:04:43 DEBUG { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "metric": { 13:04:43 DEBUG "pod": "ds-idrepo-1" 13:04:43 DEBUG "instance": "10.142.0.107:9100" 13:04:43 DEBUG }, 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG [...] 13:04:43 INFO [http_cmd]: http status code OK 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG --- status code --- 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG --- http response --- 13:04:43 DEBUG "pod": "ds-idrepo-0" 13:04:43 DEBUG "instance": "10.142.0.43:9100" 13:04:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:43 DEBUG }, 13:04:43 DEBUG }, 13:04:43 DEBUG ----- output ----- 13:04:43 DEBUG "value": [ 13:04:43 DEBUG "value": [ 13:04:43 DEBUG { 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG "status": "success", 13:04:43 DEBUG "0" 13:04:43 DEBUG "313171.8" 13:04:43 DEBUG "data": { 13:04:43 DEBUG ] 13:04:43 DEBUG ] 13:04:43 DEBUG "resultType": "vector", 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 DEBUG "result": [ 13:04:43 DEBUG ] 13:04:43 DEBUG ] 13:04:43 DEBUG { 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 DEBUG "metric": { 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:43 DEBUG "instance": "10.142.0.107:9100" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG [...] 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG 13:04:43 DEBUG "instance": "10.142.0.43:9100" 13:04:43 DEBUG }, 13:04:43 DEBUG "value": [ 13:04:43 DEBUG 1713096283, 13:04:43 DEBUG "13014907904" 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG ] 13:04:43 DEBUG } 13:04:43 DEBUG } 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096284" 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG { 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "instance": "10.142.0.107:9100" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "instance": "10.142.0.43:9100" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0.00040000000000001514" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096284" 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096284" 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096284" 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096284" 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG { 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "pod": "ds-cts-2" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG { 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "112061" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:04:44 DEBUG ] 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG } 13:04:44 DEBUG { 13:04:44 DEBUG } 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "pod": "ds-cts-2" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG { 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "pod": "ds-idrepo-1" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "pod": "ds-idrepo-0" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096284" 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096284" 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG { 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0.037" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096284" 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG { 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG { 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "712400896" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096284" 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096284" 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096284" 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG { 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "pod": "ds-idrepo-1" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "pod": "ds-idrepo-0" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG { 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG { 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096284" 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG { 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "pod": "ds-idrepo-1" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "pod": "ds-idrepo-0" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:04:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096284" 13:04:44 INFO [http_cmd]: http status code OK 13:04:44 DEBUG --- status code --- 13:04:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:44 DEBUG --- http response --- 13:04:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:44 DEBUG ----- output ----- 13:04:44 DEBUG { 13:04:44 DEBUG "status": "success", 13:04:44 DEBUG "data": { 13:04:44 DEBUG "resultType": "vector", 13:04:44 DEBUG "result": [ 13:04:44 DEBUG { 13:04:44 DEBUG "metric": { 13:04:44 DEBUG "instance": "10.142.0.107:9100" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG [...] 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG 13:04:44 DEBUG "instance": "10.142.0.43:9100" 13:04:44 DEBUG }, 13:04:44 DEBUG "value": [ 13:04:44 DEBUG 1713096284, 13:04:44 DEBUG "0" 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG ] 13:04:44 DEBUG } 13:04:44 DEBUG } 13:05:00 INFO 13:05:00 INFO [loop_until]: kubectl --namespace=xlou top pods 13:05:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:05:00 INFO [loop_until]: OK (rc = 0) 13:05:00 DEBUG --- stdout --- 13:05:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 2137m 8583Mi am-7d597cf7d6-8vr8f 1880m 8614Mi am-7d597cf7d6-9wz9h 2612m 8919Mi ds-cts-0 5154m 8090Mi ds-cts-1 5402m 8434Mi ds-cts-2 5395m 8104Mi ds-idrepo-0 1565m 4556Mi ds-idrepo-1 492m 4316Mi ds-idrepo-2 573m 4106Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 6m 1013Mi idm-b85f9d978-rl76f 6m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 786m 628Mi 13:05:00 DEBUG --- stderr --- 13:05:00 DEBUG 13:05:06 INFO 13:05:06 INFO [loop_until]: kubectl --namespace=xlou top node 13:05:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:05:07 INFO [loop_until]: OK (rc = 0) 13:05:07 DEBUG --- stdout --- 13:05:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 4874Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5513m 18% 10809Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1789m 5% 10688Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 94m 0% 3050Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 574m 1% 6567Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 4507m 15% 12031Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2952Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1811m 6% 10101Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 559m 1% 5696Mi 5% gke-xlou-cdm-ds-0f601751-j488 5013m 16% 10403Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 81m 0% 2222Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 2864m 9% 10541Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1699m 5% 6260Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 908m 5% 4233Mi 7% 13:05:07 DEBUG --- stderr --- 13:05:07 DEBUG 13:05:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096343" 13:05:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096343" 13:05:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096343" 13:05:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096343" 13:05:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096343" 13:05:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096343" 13:05:43 INFO [http_cmd]: http status code OK 13:05:43 DEBUG --- status code --- 13:05:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:43 DEBUG --- http response --- 13:05:43 INFO [http_cmd]: http status code OK 13:05:43 DEBUG --- status code --- 13:05:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:43 DEBUG --- http response --- 13:05:43 INFO [http_cmd]: http status code OK 13:05:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:05:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:05:43 DEBUG --- status code --- 13:05:43 DEBUG ----- output ----- 13:05:43 DEBUG ----- output ----- 13:05:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:43 DEBUG { 13:05:43 DEBUG { 13:05:43 DEBUG --- http response --- 13:05:43 DEBUG "status": "success", 13:05:43 DEBUG "status": "success", 13:05:43 DEBUG "data": { 13:05:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:43 DEBUG "data": { 13:05:43 DEBUG "resultType": "vector", 13:05:43 DEBUG ----- output ----- 13:05:43 DEBUG "resultType": "vector", 13:05:43 DEBUG "result": [ 13:05:43 DEBUG { 13:05:43 DEBUG "result": [ 13:05:43 DEBUG { 13:05:43 DEBUG "status": "success", 13:05:43 DEBUG { 13:05:43 DEBUG "metric": { 13:05:43 DEBUG "data": { 13:05:43 DEBUG "metric": { 13:05:43 DEBUG "pod": "ds-idrepo-1" 13:05:43 DEBUG "resultType": "vector", 13:05:43 DEBUG "pod": "ds-idrepo-1" 13:05:43 DEBUG }, 13:05:43 DEBUG "result": [ 13:05:43 DEBUG }, 13:05:43 DEBUG "value": [ 13:05:43 DEBUG { 13:05:43 DEBUG "value": [ 13:05:43 DEBUG 13:05:43 DEBUG "metric": { 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG }, 13:05:43 DEBUG 13:05:43 DEBUG [...] 13:05:43 DEBUG "value": [ 13:05:43 DEBUG [...] 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG "pod": "ds-cts-1" 13:05:43 DEBUG [...] 13:05:43 DEBUG "pod": "ds-idrepo-0" 13:05:43 DEBUG }, 13:05:43 DEBUG 13:05:43 DEBUG }, 13:05:43 DEBUG "value": [ 13:05:43 DEBUG 13:05:43 DEBUG "value": [ 13:05:43 DEBUG 1713096343, 13:05:43 DEBUG 13:05:43 DEBUG 1713096343, 13:05:43 DEBUG "0" 13:05:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:05:43 DEBUG "0" 13:05:43 DEBUG ] 13:05:43 DEBUG }, 13:05:43 DEBUG ] 13:05:43 DEBUG } 13:05:43 DEBUG "value": [ 13:05:43 DEBUG } 13:05:43 DEBUG ] 13:05:43 DEBUG 1713096343, 13:05:43 DEBUG ] 13:05:43 DEBUG } 13:05:43 DEBUG "0.6458775781090218" 13:05:43 DEBUG } 13:05:43 DEBUG } 13:05:43 DEBUG ] 13:05:43 DEBUG } 13:05:43 DEBUG } 13:05:43 DEBUG ] 13:05:43 DEBUG } 13:05:43 DEBUG } 13:05:43 INFO [http_cmd]: http status code OK 13:05:43 DEBUG --- status code --- 13:05:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:43 DEBUG --- http response --- 13:05:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:43 DEBUG ----- output ----- 13:05:43 DEBUG { 13:05:43 DEBUG "status": "success", 13:05:43 DEBUG "data": { 13:05:43 DEBUG "resultType": "vector", 13:05:43 DEBUG "result": [ 13:05:43 DEBUG { 13:05:43 DEBUG "metric": { 13:05:43 DEBUG "instance": "10.142.0.107:9100" 13:05:43 DEBUG }, 13:05:43 DEBUG "value": [ 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG [...] 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG "instance": "10.142.0.43:9100" 13:05:43 DEBUG }, 13:05:43 DEBUG "value": [ 13:05:43 DEBUG 1713096343, 13:05:43 DEBUG "6678.866666666666" 13:05:43 DEBUG ] 13:05:43 DEBUG } 13:05:43 DEBUG ] 13:05:43 DEBUG } 13:05:43 DEBUG } 13:05:43 INFO [http_cmd]: http status code OK 13:05:43 DEBUG --- status code --- 13:05:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:43 DEBUG --- http response --- 13:05:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:43 DEBUG ----- output ----- 13:05:43 DEBUG { 13:05:43 DEBUG "status": "success", 13:05:43 DEBUG "data": { 13:05:43 DEBUG "resultType": "vector", 13:05:43 DEBUG "result": [ 13:05:43 DEBUG { 13:05:43 DEBUG "metric": { 13:05:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:43 DEBUG }, 13:05:43 DEBUG "value": [ 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG [...] 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:05:43 DEBUG }, 13:05:43 DEBUG "value": [ 13:05:43 DEBUG 1713096343, 13:05:43 DEBUG "0" 13:05:43 DEBUG ] 13:05:43 DEBUG } 13:05:43 DEBUG ] 13:05:43 DEBUG } 13:05:43 DEBUG } 13:05:43 INFO [http_cmd]: http status code OK 13:05:43 DEBUG --- status code --- 13:05:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:43 DEBUG --- http response --- 13:05:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:43 DEBUG ----- output ----- 13:05:43 DEBUG { 13:05:43 DEBUG "status": "success", 13:05:43 DEBUG "data": { 13:05:43 DEBUG "resultType": "vector", 13:05:43 DEBUG "result": [ 13:05:43 DEBUG { 13:05:43 DEBUG "metric": { 13:05:43 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:43 DEBUG }, 13:05:43 DEBUG "value": [ 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG [...] 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG 13:05:43 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:05:43 DEBUG }, 13:05:43 DEBUG "value": [ 13:05:43 DEBUG 1713096343, 13:05:43 DEBUG "0" 13:05:43 DEBUG ] 13:05:43 DEBUG } 13:05:43 DEBUG ] 13:05:43 DEBUG } 13:05:43 DEBUG } 13:05:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096343" 13:05:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096343" 13:05:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096343" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096343, 13:05:44 DEBUG "279.8666666666667" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 ERROR [http_cmd]: ERROR 13:05:44 ERROR --- status code --- 13:05:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:05:44 ERROR --- http response --- 13:05:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:05:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "480765.7941421383" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096344" 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096344" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "ds-idrepo-1" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "ds-cts-1" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "ds-idrepo-1" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "ds-idrepo-0" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "instance": "10.142.0.107:9100" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "instance": "10.142.0.43:9100" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "13015434240" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "instance": "10.142.0.107:9100" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "instance": "10.142.0.43:9100" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "11962.300000000003" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096344" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "instance": "10.142.0.107:9100" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "instance": "10.142.0.43:9100" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0.0004999999999997821" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096344" 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0.1496263712388304" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "755355648" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:44 DEBUG "result": [ 13:05:44 DEBUG "result": [ 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG { 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:05:44 DEBUG "data": { 13:05:44 DEBUG }, 13:05:44 DEBUG }, 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "value": [ 13:05:44 DEBUG "value": [ 13:05:44 DEBUG "result": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG { 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "metric": { 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "ds-idrepo-1" 13:05:44 DEBUG [...] 13:05:44 DEBUG [...] 13:05:44 DEBUG }, 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:05:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:44 DEBUG 13:05:44 DEBUG }, 13:05:44 DEBUG }, 13:05:44 DEBUG [...] 13:05:44 DEBUG "value": [ 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG 13:05:44 DEBUG "4199" 13:05:44 DEBUG "325859110.67665905" 13:05:44 DEBUG 13:05:44 DEBUG ] 13:05:44 DEBUG ] 13:05:44 DEBUG "pod": "ds-idrepo-0" 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 DEBUG }, 13:05:44 DEBUG ] 13:05:44 DEBUG ] 13:05:44 DEBUG "value": [ 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 DEBUG "0" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0.037" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "ds-idrepo-1" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "ds-idrepo-0" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096344" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "pod": "ds-idrepo-1" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "pod": "ds-idrepo-0" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:05:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096344" 13:05:44 INFO [http_cmd]: http status code OK 13:05:44 DEBUG --- status code --- 13:05:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:44 DEBUG --- http response --- 13:05:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:44 DEBUG ----- output ----- 13:05:44 DEBUG { 13:05:44 DEBUG "status": "success", 13:05:44 DEBUG "data": { 13:05:44 DEBUG "resultType": "vector", 13:05:44 DEBUG "result": [ 13:05:44 DEBUG { 13:05:44 DEBUG "metric": { 13:05:44 DEBUG "instance": "10.142.0.107:9100" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG [...] 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG 13:05:44 DEBUG "instance": "10.142.0.43:9100" 13:05:44 DEBUG }, 13:05:44 DEBUG "value": [ 13:05:44 DEBUG 1713096344, 13:05:44 DEBUG "0" 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG ] 13:05:44 DEBUG } 13:05:44 DEBUG } 13:06:00 INFO 13:06:00 INFO [loop_until]: kubectl --namespace=xlou top pods 13:06:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:06:00 INFO [loop_until]: OK (rc = 0) 13:06:00 DEBUG --- stdout --- 13:06:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12m 8541Mi am-7d597cf7d6-8vr8f 7m 8613Mi am-7d597cf7d6-9wz9h 7m 8894Mi ds-cts-0 16m 8088Mi ds-cts-1 15m 8376Mi ds-cts-2 18m 8104Mi ds-idrepo-0 99m 4556Mi ds-idrepo-1 9m 4317Mi ds-idrepo-2 11m 4106Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1013Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1005m 1056Mi 13:06:00 DEBUG --- stderr --- 13:06:00 DEBUG 13:06:07 INFO 13:06:07 INFO [loop_until]: kubectl --namespace=xlou top node 13:06:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:06:07 INFO [loop_until]: OK (rc = 0) 13:06:07 DEBUG --- stdout --- 13:06:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4865Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 10776Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10646Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3034Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 6560Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 180m 0% 12009Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2951Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 10086Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 5712Mi 5% gke-xlou-cdm-ds-0f601751-j488 69m 0% 10365Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2231Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 77m 0% 10531Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 154m 0% 6217Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 4658Mi 7% 13:06:07 DEBUG --- stderr --- 13:06:07 DEBUG 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096404" 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:06:44 DEBUG { 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG { 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG "data": { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "data": { 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG "result": [ 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG { 13:06:44 DEBUG "result": [ 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:44 DEBUG "metric": { 13:06:44 DEBUG { 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG "pod": "ds-idrepo-1" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 DEBUG "metric": { 13:06:44 DEBUG { 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:44 DEBUG }, 13:06:44 DEBUG "pod": "ds-idrepo-1" 13:06:44 DEBUG "status": "success", 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 ERROR [http_cmd]: ERROR 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG "value": [ 13:06:44 DEBUG }, 13:06:44 DEBUG "data": { 13:06:44 DEBUG --- status code --- 13:06:44 ERROR --- status code --- 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG { 13:06:44 DEBUG 13:06:44 DEBUG "value": [ 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "result": [ 13:06:44 DEBUG --- http response --- 13:06:44 ERROR --- http response --- 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG "data": { 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG { 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG "metric": { 13:06:44 DEBUG ----- output ----- 13:06:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:06:44 DEBUG "result": [ 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG "instance": "10.142.0.107:9100" 13:06:44 DEBUG { 13:06:44 DEBUG { 13:06:44 DEBUG { 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG }, 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "value": [ 13:06:44 DEBUG "data": { 13:06:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:06:44 DEBUG "data": { 13:06:44 DEBUG "pod": "ds-cts-1" 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG }, 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG }, 13:06:44 DEBUG "pod": "ds-idrepo-0" 13:06:44 DEBUG 13:06:44 DEBUG "result": [ 13:06:44 DEBUG "value": [ 13:06:44 DEBUG "result": [ 13:06:44 DEBUG "value": [ 13:06:44 DEBUG }, 13:06:44 DEBUG 13:06:44 DEBUG { 13:06:44 DEBUG 13:06:44 DEBUG { 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG [...] 13:06:44 DEBUG "metric": { 13:06:44 DEBUG 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "0" 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG 13:06:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:06:44 DEBUG 13:06:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:06:44 DEBUG ] 13:06:44 DEBUG "0" 13:06:44 DEBUG 13:06:44 DEBUG }, 13:06:44 DEBUG [...] 13:06:44 DEBUG }, 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG "value": [ 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG "instance": "10.142.0.43:9100" 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG }, 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:06:44 DEBUG 13:06:44 DEBUG } 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG [...] 13:06:44 DEBUG }, 13:06:44 DEBUG [...] 13:06:44 DEBUG "5284.433333333333" 13:06:44 DEBUG 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG ] 13:06:44 DEBUG 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG 13:06:44 DEBUG } 13:06:44 DEBUG 13:06:44 DEBUG "1.0290465858818907" 13:06:44 DEBUG 13:06:44 DEBUG ] 13:06:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:06:44 DEBUG ] 13:06:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:06:44 DEBUG } 13:06:44 DEBUG }, 13:06:44 DEBUG } 13:06:44 DEBUG }, 13:06:44 DEBUG } 13:06:44 DEBUG "value": [ 13:06:44 DEBUG ] 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG } 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0" 13:06:44 DEBUG } 13:06:44 DEBUG "0" 13:06:44 DEBUG ] 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "ds-idrepo-0" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "4305.979840455136" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096404" 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "ds-idrepo-1" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "ds-cts-1" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "instance": "10.142.0.107:9100" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "instance": "10.142.0.43:9100" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "13016077312" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "ds-idrepo-1" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "ds-idrepo-0" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "instance": "10.142.0.107:9100" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "instance": "10.142.0.43:9100" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "12207.366666666665" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "instance": "10.142.0.107:9100" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "instance": "10.142.0.43:9100" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0.00043333333333350765" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "ds-idrepo-0" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0.010101787150118674" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "1206018048" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "ds-idrepo-0" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "147787813.74064684" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "ds-idrepo-1" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "ds-idrepo-0" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "1" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0.037" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "ds-idrepo-0" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "ds-idrepo-1" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "ds-idrepo-0" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096404" 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "pod": "ds-idrepo-1" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "pod": "ds-idrepo-0" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:06:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096404" 13:06:44 INFO [http_cmd]: http status code OK 13:06:44 DEBUG --- status code --- 13:06:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:44 DEBUG --- http response --- 13:06:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:44 DEBUG ----- output ----- 13:06:44 DEBUG { 13:06:44 DEBUG "status": "success", 13:06:44 DEBUG "data": { 13:06:44 DEBUG "resultType": "vector", 13:06:44 DEBUG "result": [ 13:06:44 DEBUG { 13:06:44 DEBUG "metric": { 13:06:44 DEBUG "instance": "10.142.0.107:9100" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG [...] 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG 13:06:44 DEBUG "instance": "10.142.0.43:9100" 13:06:44 DEBUG }, 13:06:44 DEBUG "value": [ 13:06:44 DEBUG 1713096404, 13:06:44 DEBUG "0.00033333333333333365" 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG ] 13:06:44 DEBUG } 13:06:44 DEBUG } 13:07:00 INFO 13:07:00 INFO [loop_until]: kubectl --namespace=xlou top pods 13:07:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:07:01 INFO [loop_until]: OK (rc = 0) 13:07:01 DEBUG --- stdout --- 13:07:01 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8531Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 7m 8893Mi ds-cts-0 16m 8089Mi ds-cts-1 15m 8375Mi ds-cts-2 16m 8105Mi ds-idrepo-0 11m 4556Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 11m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1013Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1004m 1296Mi 13:07:01 DEBUG --- stderr --- 13:07:01 DEBUG 13:07:07 INFO 13:07:07 INFO [loop_until]: kubectl --namespace=xlou top node 13:07:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:07:07 INFO [loop_until]: OK (rc = 0) 13:07:07 DEBUG --- stdout --- 13:07:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4878Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 10770Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 10634Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3032Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 65m 0% 6578Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 12027Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2950Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 10078Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5735Mi 5% gke-xlou-cdm-ds-0f601751-j488 68m 0% 10372Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 68m 0% 2228Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 86m 0% 10517Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 65m 0% 6223Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1019m 6% 4670Mi 7% 13:07:07 DEBUG --- stderr --- 13:07:07 DEBUG 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 ERROR [http_cmd]: ERROR 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 ERROR --- status code --- 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG --- status code --- 13:07:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 ERROR --- http response --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG { 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "data": { 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG "result": [ 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG { 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "metric": { 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG "metric": { 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG "pod": "ds-idrepo-1" 13:07:44 DEBUG "instance": "10.142.0.107:9100" 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG }, 13:07:44 DEBUG }, 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG }, 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG "value": [ 13:07:44 DEBUG "value": [ 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG "value": [ 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:44 DEBUG 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG { 13:07:44 DEBUG [...] 13:07:44 DEBUG { 13:07:44 DEBUG [...] 13:07:44 DEBUG [...] 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "data": { 13:07:44 DEBUG 13:07:44 DEBUG "data": { 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "result": [ 13:07:44 DEBUG "pod": "ds-cts-1" 13:07:44 DEBUG "result": [ 13:07:44 DEBUG "instance": "10.142.0.43:9100" 13:07:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:07:44 DEBUG { 13:07:44 DEBUG }, 13:07:44 DEBUG { 13:07:44 DEBUG }, 13:07:44 DEBUG }, 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "value": [ 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "value": [ 13:07:44 DEBUG "value": [ 13:07:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "pod": "ds-idrepo-1" 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG }, 13:07:44 DEBUG "0" 13:07:44 DEBUG }, 13:07:44 DEBUG "5455.166666666666" 13:07:44 DEBUG "0.6707264385769874" 13:07:44 DEBUG "value": [ 13:07:44 DEBUG ] 13:07:44 DEBUG "value": [ 13:07:44 DEBUG ] 13:07:44 DEBUG ] 13:07:44 DEBUG 13:07:44 DEBUG } 13:07:44 DEBUG 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 DEBUG 13:07:44 DEBUG ] 13:07:44 DEBUG 13:07:44 DEBUG ] 13:07:44 DEBUG ] 13:07:44 DEBUG 13:07:44 DEBUG } 13:07:44 DEBUG 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 DEBUG [...] 13:07:44 DEBUG } 13:07:44 DEBUG [...] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:07:44 DEBUG "pod": "ds-idrepo-0" 13:07:44 DEBUG }, 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0" 13:07:44 DEBUG "0" 13:07:44 DEBUG ] 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG } 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "ds-idrepo-0" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "4635.549915977915" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096464" 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "pod": "ds-idrepo-1" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "ds-cts-1" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "pod": "ds-idrepo-1" 13:07:44 DEBUG "instance": "10.142.0.107:9100" 13:07:44 DEBUG }, 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "ds-idrepo-0" 13:07:44 DEBUG "instance": "10.142.0.43:9100" 13:07:44 DEBUG }, 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0" 13:07:44 DEBUG "13016699904" 13:07:44 DEBUG ] 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "instance": "10.142.0.107:9100" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "instance": "10.142.0.43:9100" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "12331.733333333332" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "instance": "10.142.0.107:9100" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "instance": "10.142.0.43:9100" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0.000500000000000019" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG { 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG --- http response --- 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG { 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG "data": { 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG "data": { 13:07:44 DEBUG { 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG "result": [ 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG { 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:07:44 DEBUG "result": [ 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG "data": { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "pod": "ds-idrepo-1" 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG }, 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:07:44 DEBUG "data": { 13:07:44 DEBUG { 13:07:44 DEBUG "value": [ 13:07:44 DEBUG "data": { 13:07:44 DEBUG }, 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "metric": { 13:07:44 DEBUG 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "value": [ 13:07:44 DEBUG "result": [ 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:07:44 DEBUG 13:07:44 DEBUG "result": [ 13:07:44 DEBUG 13:07:44 DEBUG { 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG }, 13:07:44 DEBUG 13:07:44 DEBUG { 13:07:44 DEBUG 13:07:44 DEBUG "metric": { 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG "value": [ 13:07:44 DEBUG [...] 13:07:44 DEBUG "metric": { 13:07:44 DEBUG 13:07:44 DEBUG "pod": "ds-idrepo-1" 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:07:44 DEBUG [...] 13:07:44 DEBUG }, 13:07:44 DEBUG 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:44 DEBUG 13:07:44 DEBUG }, 13:07:44 DEBUG 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG { 13:07:44 DEBUG "pod": "ds-idrepo-0" 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG }, 13:07:44 DEBUG 13:07:44 DEBUG "pod": "ds-idrepo-0" 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "data": { 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG }, 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG [...] 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:07:44 DEBUG "result": [ 13:07:44 DEBUG "0" 13:07:44 DEBUG 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG 13:07:44 DEBUG }, 13:07:44 DEBUG { 13:07:44 DEBUG ] 13:07:44 DEBUG 13:07:44 DEBUG "0.010375561576591884" 13:07:44 DEBUG 13:07:44 DEBUG "value": [ 13:07:44 DEBUG "metric": { 13:07:44 DEBUG } 13:07:44 DEBUG 13:07:44 DEBUG ] 13:07:44 DEBUG "pod": "ds-idrepo-0" 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:07:44 DEBUG ] 13:07:44 DEBUG "pod": "ds-idrepo-0" 13:07:44 DEBUG } 13:07:44 DEBUG }, 13:07:44 DEBUG "1212841984" 13:07:44 DEBUG }, 13:07:44 DEBUG } 13:07:44 DEBUG }, 13:07:44 DEBUG ] 13:07:44 DEBUG "value": [ 13:07:44 DEBUG ] 13:07:44 DEBUG "value": [ 13:07:44 DEBUG } 13:07:44 DEBUG "value": [ 13:07:44 DEBUG } 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG } 13:07:44 DEBUG 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG } 13:07:44 DEBUG "0" 13:07:44 DEBUG ] 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 DEBUG 13:07:44 DEBUG "18963.613292636925" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 DEBUG [...] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 DEBUG 13:07:44 DEBUG } 13:07:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "1" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0.037" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "ds-idrepo-0" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096464" 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "pod": "ds-idrepo-1" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "pod": "ds-idrepo-0" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:07:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096464" 13:07:44 INFO [http_cmd]: http status code OK 13:07:44 DEBUG --- status code --- 13:07:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:44 DEBUG --- http response --- 13:07:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:44 DEBUG ----- output ----- 13:07:44 DEBUG { 13:07:44 DEBUG "status": "success", 13:07:44 DEBUG "data": { 13:07:44 DEBUG "resultType": "vector", 13:07:44 DEBUG "result": [ 13:07:44 DEBUG { 13:07:44 DEBUG "metric": { 13:07:44 DEBUG "instance": "10.142.0.107:9100" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG [...] 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG 13:07:44 DEBUG "instance": "10.142.0.43:9100" 13:07:44 DEBUG }, 13:07:44 DEBUG "value": [ 13:07:44 DEBUG 1713096464, 13:07:44 DEBUG "0.0006666666666666673" 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG ] 13:07:44 DEBUG } 13:07:44 DEBUG } 13:08:01 INFO 13:08:01 INFO [loop_until]: kubectl --namespace=xlou top pods 13:08:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:08:01 INFO [loop_until]: OK (rc = 0) 13:08:01 DEBUG --- stdout --- 13:08:01 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8532Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 7m 8893Mi ds-cts-0 17m 8088Mi ds-cts-1 15m 8375Mi ds-cts-2 16m 8104Mi ds-idrepo-0 12m 4556Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 10m 4106Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 10m 1013Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1065m 1436Mi 13:08:01 DEBUG --- stderr --- 13:08:01 DEBUG 13:08:07 INFO 13:08:07 INFO [loop_until]: kubectl --namespace=xlou top node 13:08:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:08:07 INFO [loop_until]: OK (rc = 0) 13:08:07 DEBUG --- stdout --- 13:08:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 59m 0% 4876Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 10764Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10631Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3038Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 64m 0% 6588Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 12051Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 70m 0% 2949Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 58m 0% 10075Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 5735Mi 5% gke-xlou-cdm-ds-0f601751-j488 71m 0% 10357Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2221Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 82m 0% 10503Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 6222Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1117m 7% 5049Mi 8% 13:08:07 DEBUG --- stderr --- 13:08:07 DEBUG 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 ERROR [http_cmd]: ERROR 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 ERROR --- status code --- 13:08:44 DEBUG --- status code --- 13:08:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 ERROR --- http response --- 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:08:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "pod": "ds-idrepo-1" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "ds-idrepo-0" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG { 13:08:44 DEBUG { 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG "data": { 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG "data": { 13:08:44 DEBUG "data": { 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG "result": [ 13:08:44 DEBUG "result": [ 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG "result": [ 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:44 DEBUG { 13:08:44 DEBUG { 13:08:44 DEBUG { 13:08:44 DEBUG { 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "metric": { 13:08:44 DEBUG { 13:08:44 DEBUG "pod": "ds-idrepo-1" 13:08:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:08:44 DEBUG "data": { 13:08:44 DEBUG "instance": "10.142.0.107:9100" 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG }, 13:08:44 DEBUG }, 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG }, 13:08:44 DEBUG "data": { 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG "value": [ 13:08:44 DEBUG "value": [ 13:08:44 DEBUG "result": [ 13:08:44 DEBUG "value": [ 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG { 13:08:44 DEBUG 13:08:44 DEBUG "result": [ 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "metric": { 13:08:44 DEBUG 13:08:44 DEBUG { 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:08:44 DEBUG 13:08:44 DEBUG "metric": { 13:08:44 DEBUG [...] 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:44 DEBUG [...] 13:08:44 DEBUG }, 13:08:44 DEBUG [...] 13:08:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:08:44 DEBUG 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG }, 13:08:44 DEBUG 13:08:44 DEBUG { 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "ds-cts-1" 13:08:44 DEBUG "data": { 13:08:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:08:44 DEBUG 13:08:44 DEBUG "instance": "10.142.0.43:9100" 13:08:44 DEBUG 13:08:44 DEBUG }, 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG }, 13:08:44 DEBUG [...] 13:08:44 DEBUG }, 13:08:44 DEBUG 13:08:44 DEBUG "value": [ 13:08:44 DEBUG "result": [ 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG "value": [ 13:08:44 DEBUG [...] 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG { 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG 13:08:44 DEBUG "0" 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "0" 13:08:44 DEBUG 13:08:44 DEBUG "5118.066666666667" 13:08:44 DEBUG 13:08:44 DEBUG ] 13:08:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:08:44 DEBUG ] 13:08:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:08:44 DEBUG ] 13:08:44 DEBUG 13:08:44 DEBUG } 13:08:44 DEBUG }, 13:08:44 DEBUG } 13:08:44 DEBUG }, 13:08:44 DEBUG } 13:08:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:08:44 DEBUG ] 13:08:44 DEBUG "value": [ 13:08:44 DEBUG ] 13:08:44 DEBUG "value": [ 13:08:44 DEBUG ] 13:08:44 DEBUG }, 13:08:44 DEBUG } 13:08:44 DEBUG 13:08:44 DEBUG } 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG } 13:08:44 DEBUG "value": [ 13:08:44 DEBUG } 13:08:44 DEBUG 13:08:44 DEBUG } 13:08:44 DEBUG "0" 13:08:44 DEBUG } 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG 13:08:44 DEBUG ] 13:08:44 DEBUG "1.0651661684046263" 13:08:44 DEBUG [...] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:08:44 DEBUG } 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096524" 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "instance": "10.142.0.107:9100" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "instance": "10.142.0.43:9100" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "13017373696" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "pod": "ds-idrepo-1" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "ds-cts-1" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "pod": "ds-idrepo-1" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "ds-idrepo-0" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "instance": "10.142.0.107:9100" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "instance": "10.142.0.43:9100" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "12242.666666666664" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096524" 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "instance": "10.142.0.107:9100" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "instance": "10.142.0.43:9100" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0.00040000000000001514" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG { 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:08:44 DEBUG { 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG "data": { 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "data": { 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG "result": [ 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "result": [ 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:44 DEBUG { 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:44 DEBUG "data": { 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG { 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG "metric": { 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG "result": [ 13:08:44 DEBUG "metric": { 13:08:44 DEBUG { 13:08:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:08:44 DEBUG { 13:08:44 DEBUG "result": [ 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG { 13:08:44 DEBUG "pod": "ds-idrepo-1" 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG }, 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG { 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG "metric": { 13:08:44 DEBUG }, 13:08:44 DEBUG "data": { 13:08:44 DEBUG "value": [ 13:08:44 DEBUG "data": { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:44 DEBUG "value": [ 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "pod": "ds-idrepo-1" 13:08:44 DEBUG }, 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG 13:08:44 DEBUG "result": [ 13:08:44 DEBUG 13:08:44 DEBUG "result": [ 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG { 13:08:44 DEBUG 13:08:44 DEBUG { 13:08:44 DEBUG 13:08:44 DEBUG { 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG 13:08:44 DEBUG "metric": { 13:08:44 DEBUG [...] 13:08:44 DEBUG "metric": { 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "data": { 13:08:44 DEBUG [...] 13:08:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:08:44 DEBUG 13:08:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG 13:08:44 DEBUG }, 13:08:44 DEBUG 13:08:44 DEBUG }, 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG "result": [ 13:08:44 DEBUG 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG "value": [ 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG { 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "pod": "ds-idrepo-0" 13:08:44 DEBUG 13:08:44 DEBUG }, 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:08:44 DEBUG }, 13:08:44 DEBUG 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "idm-b85f9d978-rl76f" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG [...] 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG [...] 13:08:44 DEBUG "pod": "ds-idrepo-0" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG 13:08:44 DEBUG "0" 13:08:44 DEBUG 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG "0" 13:08:44 DEBUG 13:08:44 DEBUG ] 13:08:44 DEBUG 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG 13:08:44 DEBUG ] 13:08:44 DEBUG 13:08:44 DEBUG } 13:08:44 DEBUG 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0" 13:08:44 DEBUG 13:08:44 DEBUG } 13:08:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:08:44 DEBUG ] 13:08:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:08:44 DEBUG "0" 13:08:44 DEBUG ] 13:08:44 DEBUG [...] 13:08:44 DEBUG ] 13:08:44 DEBUG }, 13:08:44 DEBUG } 13:08:44 DEBUG }, 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG 13:08:44 DEBUG } 13:08:44 DEBUG "value": [ 13:08:44 DEBUG } 13:08:44 DEBUG "value": [ 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG 13:08:44 DEBUG } 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG 13:08:44 DEBUG "0" 13:08:44 DEBUG "1605799936" 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:08:44 DEBUG ] 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG }, 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 DEBUG "value": [ 13:08:44 DEBUG ] 13:08:44 DEBUG ] 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 DEBUG "0.037" 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "idm-b85f9d978-rl76f" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096524" 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "pod": "ds-idrepo-1" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "pod": "ds-idrepo-0" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:08:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096524" 13:08:44 INFO [http_cmd]: http status code OK 13:08:44 DEBUG --- status code --- 13:08:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:44 DEBUG --- http response --- 13:08:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:44 DEBUG ----- output ----- 13:08:44 DEBUG { 13:08:44 DEBUG "status": "success", 13:08:44 DEBUG "data": { 13:08:44 DEBUG "resultType": "vector", 13:08:44 DEBUG "result": [ 13:08:44 DEBUG { 13:08:44 DEBUG "metric": { 13:08:44 DEBUG "instance": "10.142.0.107:9100" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG [...] 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG 13:08:44 DEBUG "instance": "10.142.0.43:9100" 13:08:44 DEBUG }, 13:08:44 DEBUG "value": [ 13:08:44 DEBUG 1713096524, 13:08:44 DEBUG "0.00033333333333333365" 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG ] 13:08:44 DEBUG } 13:08:44 DEBUG } 13:09:01 INFO 13:09:01 INFO [loop_until]: kubectl --namespace=xlou top pods 13:09:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:09:01 INFO [loop_until]: OK (rc = 0) 13:09:01 DEBUG --- stdout --- 13:09:01 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8532Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 7m 8892Mi ds-cts-0 17m 8088Mi ds-cts-1 15m 8376Mi ds-cts-2 20m 8104Mi ds-idrepo-0 11m 4556Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 12m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1013Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 1m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1007m 1813Mi 13:09:01 DEBUG --- stderr --- 13:09:01 DEBUG 13:09:07 INFO 13:09:07 INFO [loop_until]: kubectl --namespace=xlou top node 13:09:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:09:07 INFO [loop_until]: OK (rc = 0) 13:09:07 DEBUG --- stdout --- 13:09:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4876Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 75m 0% 10765Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 10614Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3047Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 65m 0% 6595Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 12038Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2946Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 10075Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5730Mi 5% gke-xlou-cdm-ds-0f601751-j488 71m 0% 10362Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 68m 0% 2229Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 84m 0% 10478Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 61m 0% 6236Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1040m 6% 5451Mi 9% 13:09:07 DEBUG --- stderr --- 13:09:07 DEBUG 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 ERROR [http_cmd]: ERROR 13:09:44 ERROR --- status code --- 13:09:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:09:44 ERROR --- http response --- 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:09:44 DEBUG --- status code --- 13:09:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "ds-idrepo-1" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "ds-idrepo-0" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "1.028188286186457" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:09:44 DEBUG "pod": "ds-idrepo-1" 13:09:44 DEBUG }, 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:09:44 DEBUG "pod": "ds-cts-1" 13:09:44 DEBUG }, 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096584" 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "instance": "10.142.0.107:9100" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "instance": "10.142.0.43:9100" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "6943.8" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096584" 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "instance": "10.142.0.107:9100" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "instance": "10.142.0.43:9100" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "13017893888" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096584" 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "ds-idrepo-1" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "ds-cts-1" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "ds-idrepo-1" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "ds-idrepo-0" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "instance": "10.142.0.107:9100" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "instance": "10.142.0.43:9100" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "12078.233333333332" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096584" 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "instance": "10.142.0.107:9100" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "instance": "10.142.0.43:9100" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0.00036666666666675953" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096584" 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "ds-idrepo-1" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "ds-idrepo-0" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "13706314.4558648" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 DEBUG ----- output ----- 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "2000175104" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "ds-idrepo-1" 13:09:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:09:44 DEBUG }, 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "ds-idrepo-0" 13:09:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:09:44 DEBUG }, 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG 1713096584, 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG "0" 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG "0" 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG ] 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG ] 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG } 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG } 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG ] 13:09:44 DEBUG ] 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:44 DEBUG } 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG } 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG } 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG { 13:09:44 DEBUG { 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG "data": { 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG { 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:09:44 DEBUG "data": { 13:09:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:09:44 DEBUG }, 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG "result": [ 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG { 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "metric": { 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG }, 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:09:44 DEBUG 13:09:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:09:44 DEBUG }, 13:09:44 DEBUG [...] 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG 13:09:44 DEBUG "2.005995623117946" 13:09:44 DEBUG ] 13:09:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG }, 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG "value": [ 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 DEBUG "0.037" 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096584" 13:09:44 INFO [http_cmd]: http status code OK 13:09:44 DEBUG --- status code --- 13:09:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:44 DEBUG --- http response --- 13:09:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:44 DEBUG ----- output ----- 13:09:44 DEBUG { 13:09:44 DEBUG "status": "success", 13:09:44 DEBUG "data": { 13:09:44 DEBUG "resultType": "vector", 13:09:44 DEBUG "result": [ 13:09:44 DEBUG { 13:09:44 DEBUG "metric": { 13:09:44 DEBUG "pod": "ds-idrepo-1" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG [...] 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG 13:09:44 DEBUG "pod": "ds-idrepo-0" 13:09:44 DEBUG }, 13:09:44 DEBUG "value": [ 13:09:44 DEBUG 1713096584, 13:09:44 DEBUG "0" 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG ] 13:09:44 DEBUG } 13:09:44 DEBUG } 13:09:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096584" 13:09:45 INFO [http_cmd]: http status code OK 13:09:45 DEBUG --- status code --- 13:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:45 DEBUG --- http response --- 13:09:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:45 DEBUG ----- output ----- 13:09:45 DEBUG { 13:09:45 DEBUG "status": "success", 13:09:45 DEBUG "data": { 13:09:45 DEBUG "resultType": "vector", 13:09:45 DEBUG "result": [ 13:09:45 DEBUG { 13:09:45 DEBUG "metric": { 13:09:45 DEBUG "instance": "10.142.0.107:9100" 13:09:45 DEBUG }, 13:09:45 DEBUG "value": [ 13:09:45 DEBUG 13:09:45 DEBUG 13:09:45 DEBUG 13:09:45 DEBUG [...] 13:09:45 DEBUG 13:09:45 DEBUG 13:09:45 DEBUG 13:09:45 DEBUG "instance": "10.142.0.43:9100" 13:09:45 DEBUG }, 13:09:45 DEBUG "value": [ 13:09:45 DEBUG 1713096584, 13:09:45 DEBUG "0.0003333333333333299" 13:09:45 DEBUG ] 13:09:45 DEBUG } 13:09:45 DEBUG ] 13:09:45 DEBUG } 13:09:45 DEBUG } 13:10:01 INFO 13:10:01 INFO [loop_until]: kubectl --namespace=xlou top pods 13:10:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:10:01 INFO [loop_until]: OK (rc = 0) 13:10:01 DEBUG --- stdout --- 13:10:01 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8531Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 6m 8893Mi ds-cts-0 18m 8089Mi ds-cts-1 18m 8376Mi ds-cts-2 17m 8104Mi ds-idrepo-0 11m 4557Mi ds-idrepo-1 44m 4317Mi ds-idrepo-2 10m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1013Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1012m 2167Mi 13:10:01 DEBUG --- stderr --- 13:10:01 DEBUG 13:10:07 INFO 13:10:07 INFO [loop_until]: kubectl --namespace=xlou top node 13:10:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:10:07 INFO [loop_until]: OK (rc = 0) 13:10:07 DEBUG --- stdout --- 13:10:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4877Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 74m 0% 10764Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10611Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3049Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 106m 0% 6597Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 12058Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2940Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 59m 0% 10073Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 5727Mi 5% gke-xlou-cdm-ds-0f601751-j488 67m 0% 10361Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2230Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 86m 0% 10482Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 67m 0% 6227Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 5846Mi 9% 13:10:07 DEBUG --- stderr --- 13:10:07 DEBUG 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "instance": "10.142.0.107:9100" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "instance": "10.142.0.43:9100" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "5329.7" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG --- status code --- 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 ERROR [http_cmd]: ERROR 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- status code --- 13:10:44 ERROR --- status code --- 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:10:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:44 ERROR --- http response --- 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "ds-idrepo-1" 13:10:44 DEBUG }, 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "ds-idrepo-0" 13:10:44 DEBUG }, 13:10:44 DEBUG 13:10:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG "1.0321325025127621" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "ds-idrepo-1" 13:10:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:10:44 DEBUG }, 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "ds-cts-1" 13:10:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:10:44 DEBUG }, 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG "1626.3704507577224" 13:10:44 DEBUG ] 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "ds-idrepo-1" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "ds-cts-1" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "instance": "10.142.0.107:9100" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "instance": "10.142.0.43:9100" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "13018577920" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "ds-idrepo-1" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "ds-idrepo-0" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "instance": "10.142.0.107:9100" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "instance": "10.142.0.43:9100" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "12068.466666666665" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "instance": "10.142.0.107:9100" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "instance": "10.142.0.43:9100" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0.0004333333333332708" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "ds-idrepo-1" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "ds-idrepo-0" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "ds-idrepo-0" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "ds-idrepo-1" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "ds-idrepo-0" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "2480599040" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0.037" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "ds-idrepo-0" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096644" 13:10:44 INFO [http_cmd]: http status code OK 13:10:44 DEBUG --- status code --- 13:10:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:44 DEBUG --- http response --- 13:10:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:44 DEBUG ----- output ----- 13:10:44 DEBUG { 13:10:44 DEBUG "status": "success", 13:10:44 DEBUG "data": { 13:10:44 DEBUG "resultType": "vector", 13:10:44 DEBUG "result": [ 13:10:44 DEBUG { 13:10:44 DEBUG "metric": { 13:10:44 DEBUG "pod": "ds-idrepo-1" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG [...] 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG 13:10:44 DEBUG "pod": "ds-idrepo-0" 13:10:44 DEBUG }, 13:10:44 DEBUG "value": [ 13:10:44 DEBUG 1713096644, 13:10:44 DEBUG "0" 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG ] 13:10:44 DEBUG } 13:10:44 DEBUG } 13:10:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096645" 13:10:45 INFO [http_cmd]: http status code OK 13:10:45 DEBUG --- status code --- 13:10:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:45 DEBUG --- http response --- 13:10:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:45 DEBUG ----- output ----- 13:10:45 DEBUG { 13:10:45 DEBUG "status": "success", 13:10:45 DEBUG "data": { 13:10:45 DEBUG "resultType": "vector", 13:10:45 DEBUG "result": [ 13:10:45 DEBUG { 13:10:45 DEBUG "metric": { 13:10:45 DEBUG "instance": "10.142.0.107:9100" 13:10:45 DEBUG }, 13:10:45 DEBUG "value": [ 13:10:45 DEBUG 13:10:45 DEBUG 13:10:45 DEBUG 13:10:45 DEBUG [...] 13:10:45 DEBUG 13:10:45 DEBUG 13:10:45 DEBUG 13:10:45 DEBUG "instance": "10.142.0.43:9100" 13:10:45 DEBUG }, 13:10:45 DEBUG "value": [ 13:10:45 DEBUG 1713096645, 13:10:45 DEBUG "0.00033333333333333365" 13:10:45 DEBUG ] 13:10:45 DEBUG } 13:10:45 DEBUG ] 13:10:45 DEBUG } 13:10:45 DEBUG } 13:11:01 INFO 13:11:01 INFO [loop_until]: kubectl --namespace=xlou top pods 13:11:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:11:01 INFO [loop_until]: OK (rc = 0) 13:11:01 DEBUG --- stdout --- 13:11:01 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8531Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 7m 8892Mi ds-cts-0 18m 8089Mi ds-cts-1 16m 8375Mi ds-cts-2 17m 8104Mi ds-idrepo-0 12m 4556Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 13m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1013Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 916m 2329Mi 13:11:01 DEBUG --- stderr --- 13:11:01 DEBUG 13:11:07 INFO 13:11:07 INFO [loop_until]: kubectl --namespace=xlou top node 13:11:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:11:07 INFO [loop_until]: OK (rc = 0) 13:11:07 DEBUG --- stdout --- 13:11:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4873Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 10756Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 51m 0% 10619Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 91m 0% 3056Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 66m 0% 6589Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 12069Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2944Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 56m 0% 10066Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 66m 0% 5724Mi 5% gke-xlou-cdm-ds-0f601751-j488 67m 0% 10357Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2225Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 86m 0% 10473Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 62m 0% 6233Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 995m 6% 5969Mi 10% 13:11:07 DEBUG --- stderr --- 13:11:07 DEBUG 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 ERROR [http_cmd]: ERROR 13:11:44 ERROR --- status code --- 13:11:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:11:44 ERROR --- http response --- 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "instance": "10.142.0.107:9100" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "instance": "10.142.0.43:9100" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "5582.733333333333" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-idrepo-0" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "4800.996300501813" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0.9052409902325388" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG { 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "ds-idrepo-1" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-idrepo-0" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "ds-idrepo-1" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-cts-1" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "ds-idrepo-1" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-cts-1" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "instance": "10.142.0.107:9100" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "instance": "10.142.0.43:9100" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "13019210752" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG "status": "success", 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG "data": { 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG "result": [ 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG { 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:11:44 DEBUG "metric": { 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG }, 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "value": [ 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG [...] 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "ds-idrepo-1" 13:11:44 DEBUG 13:11:44 DEBUG "instance": "10.142.0.107:9100" 13:11:44 DEBUG }, 13:11:44 DEBUG 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG }, 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG "0" 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG ] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG } 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG ] 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-idrepo-0" 13:11:44 DEBUG } 13:11:44 DEBUG "instance": "10.142.0.43:9100" 13:11:44 DEBUG }, 13:11:44 DEBUG } 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG "12008.366666666665" 13:11:44 DEBUG ] 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "instance": "10.142.0.107:9100" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "instance": "10.142.0.43:9100" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0.00043333333333350765" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-idrepo-0" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "ds-idrepo-1" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-idrepo-0" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "2540199936" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-idrepo-0" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "175093835.390645" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-idrepo-0" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0.010746273030977058" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "ds-idrepo-1" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-idrepo-0" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0.037" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096704" 13:11:44 INFO [http_cmd]: http status code OK 13:11:44 DEBUG --- status code --- 13:11:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:44 DEBUG --- http response --- 13:11:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:11:44 DEBUG ----- output ----- 13:11:44 DEBUG { 13:11:44 DEBUG "status": "success", 13:11:44 DEBUG "data": { 13:11:44 DEBUG "resultType": "vector", 13:11:44 DEBUG "result": [ 13:11:44 DEBUG { 13:11:44 DEBUG "metric": { 13:11:44 DEBUG "pod": "ds-idrepo-1" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG [...] 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG 13:11:44 DEBUG "pod": "ds-idrepo-0" 13:11:44 DEBUG }, 13:11:44 DEBUG "value": [ 13:11:44 DEBUG 1713096704, 13:11:44 DEBUG "0" 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG ] 13:11:44 DEBUG } 13:11:44 DEBUG } 13:11:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096705" 13:11:45 INFO [http_cmd]: http status code OK 13:11:45 DEBUG --- status code --- 13:11:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:11:45 DEBUG --- http response --- 13:11:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:11:45 DEBUG ----- output ----- 13:11:45 DEBUG { 13:11:45 DEBUG "status": "success", 13:11:45 DEBUG "data": { 13:11:45 DEBUG "resultType": "vector", 13:11:45 DEBUG "result": [ 13:11:45 DEBUG { 13:11:45 DEBUG "metric": { 13:11:45 DEBUG "instance": "10.142.0.107:9100" 13:11:45 DEBUG }, 13:11:45 DEBUG "value": [ 13:11:45 DEBUG 13:11:45 DEBUG 13:11:45 DEBUG 13:11:45 DEBUG [...] 13:11:45 DEBUG 13:11:45 DEBUG 13:11:45 DEBUG 13:11:45 DEBUG "instance": "10.142.0.43:9100" 13:11:45 DEBUG }, 13:11:45 DEBUG "value": [ 13:11:45 DEBUG 1713096705, 13:11:45 DEBUG "0" 13:11:45 DEBUG ] 13:11:45 DEBUG } 13:11:45 DEBUG ] 13:11:45 DEBUG } 13:11:45 DEBUG } 13:12:01 INFO 13:12:01 INFO [loop_until]: kubectl --namespace=xlou top pods 13:12:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:12:01 INFO [loop_until]: OK (rc = 0) 13:12:01 DEBUG --- stdout --- 13:12:01 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8531Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 7m 8893Mi ds-cts-0 18m 8089Mi ds-cts-1 17m 8375Mi ds-cts-2 18m 8105Mi ds-idrepo-0 11m 4556Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 10m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1013Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1054m 2692Mi 13:12:01 DEBUG --- stderr --- 13:12:01 DEBUG 13:12:07 INFO 13:12:07 INFO [loop_until]: kubectl --namespace=xlou top node 13:12:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:12:07 INFO [loop_until]: OK (rc = 0) 13:12:07 DEBUG --- stdout --- 13:12:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4886Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 77m 0% 10751Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 51m 0% 10607Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3043Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 63m 0% 6602Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 169m 0% 12089Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2949Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 60m 0% 10053Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5714Mi 5% gke-xlou-cdm-ds-0f601751-j488 69m 0% 10361Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2223Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 81m 0% 10479Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 62m 0% 6233Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1134m 7% 6369Mi 10% 13:12:07 DEBUG --- stderr --- 13:12:07 DEBUG 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "instance": "10.142.0.107:9100" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "instance": "10.142.0.43:9100" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "5278.633333333333" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 ERROR [http_cmd]: ERROR 13:12:44 ERROR --- status code --- 13:12:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:12:44 ERROR --- http response --- 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:12:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG "result": [ 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG { 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:12:44 DEBUG 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG [...] 13:12:44 DEBUG { 13:12:44 DEBUG 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG 13:12:44 DEBUG "data": { 13:12:44 DEBUG 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "pod": "ds-idrepo-2" 13:12:44 DEBUG "result": [ 13:12:44 DEBUG }, 13:12:44 DEBUG { 13:12:44 DEBUG "value": [ 13:12:44 DEBUG "metric": { 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:12:44 DEBUG "5035.770739418068" 13:12:44 DEBUG }, 13:12:44 DEBUG ] 13:12:44 DEBUG "value": [ 13:12:44 DEBUG } 13:12:44 DEBUG 13:12:44 DEBUG ] 13:12:44 DEBUG 13:12:44 DEBUG } 13:12:44 DEBUG 13:12:44 DEBUG } 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "1.0953169076923173" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "ds-idrepo-1" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "ds-idrepo-0" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "ds-idrepo-1" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "ds-cts-1" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "ds-idrepo-1" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "ds-cts-1" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "instance": "10.142.0.107:9100" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "instance": "10.142.0.43:9100" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "13019835392" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "instance": "10.142.0.107:9100" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "instance": "10.142.0.43:9100" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "11889.466666666664" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "ds-idrepo-1" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "ds-idrepo-0" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "instance": "10.142.0.107:9100" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "instance": "10.142.0.43:9100" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0.00040000000000001514" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "ds-idrepo-2" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "ds-idrepo-1" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "ds-idrepo-0" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "2922094592" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "ds-idrepo-0" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "ds-idrepo-0" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "ds-idrepo-1" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "ds-idrepo-0" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0.037" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096764" 13:12:44 INFO [http_cmd]: http status code OK 13:12:44 DEBUG --- status code --- 13:12:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:44 DEBUG --- http response --- 13:12:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:12:44 DEBUG ----- output ----- 13:12:44 DEBUG { 13:12:44 DEBUG "status": "success", 13:12:44 DEBUG "data": { 13:12:44 DEBUG "resultType": "vector", 13:12:44 DEBUG "result": [ 13:12:44 DEBUG { 13:12:44 DEBUG "metric": { 13:12:44 DEBUG "pod": "ds-idrepo-1" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG [...] 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG 13:12:44 DEBUG "pod": "ds-idrepo-0" 13:12:44 DEBUG }, 13:12:44 DEBUG "value": [ 13:12:44 DEBUG 1713096764, 13:12:44 DEBUG "0" 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG ] 13:12:44 DEBUG } 13:12:44 DEBUG } 13:12:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096765" 13:12:45 INFO [http_cmd]: http status code OK 13:12:45 DEBUG --- status code --- 13:12:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:12:45 DEBUG --- http response --- 13:12:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:12:45 DEBUG ----- output ----- 13:12:45 DEBUG { 13:12:45 DEBUG "status": "success", 13:12:45 DEBUG "data": { 13:12:45 DEBUG "resultType": "vector", 13:12:45 DEBUG "result": [ 13:12:45 DEBUG { 13:12:45 DEBUG "metric": { 13:12:45 DEBUG "instance": "10.142.0.107:9100" 13:12:45 DEBUG }, 13:12:45 DEBUG "value": [ 13:12:45 DEBUG 13:12:45 DEBUG 13:12:45 DEBUG 13:12:45 DEBUG [...] 13:12:45 DEBUG 13:12:45 DEBUG 13:12:45 DEBUG 13:12:45 DEBUG "instance": "10.142.0.43:9100" 13:12:45 DEBUG }, 13:12:45 DEBUG "value": [ 13:12:45 DEBUG 1713096765, 13:12:45 DEBUG "0" 13:12:45 DEBUG ] 13:12:45 DEBUG } 13:12:45 DEBUG ] 13:12:45 DEBUG } 13:12:45 DEBUG } 13:13:01 INFO 13:13:01 INFO [loop_until]: kubectl --namespace=xlou top pods 13:13:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:13:01 INFO [loop_until]: OK (rc = 0) 13:13:01 DEBUG --- stdout --- 13:13:01 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8531Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 6m 8892Mi ds-cts-0 15m 8088Mi ds-cts-1 16m 8376Mi ds-cts-2 16m 8105Mi ds-idrepo-0 11m 4555Mi ds-idrepo-1 9m 4317Mi ds-idrepo-2 11m 4106Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1007m 3176Mi 13:13:01 DEBUG --- stderr --- 13:13:01 DEBUG 13:13:07 INFO 13:13:07 INFO [loop_until]: kubectl --namespace=xlou top node 13:13:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:13:08 INFO [loop_until]: OK (rc = 0) 13:13:08 DEBUG --- stdout --- 13:13:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4872Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 10747Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 10608Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 6597Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 12072Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2938Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 58m 0% 10065Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5721Mi 5% gke-xlou-cdm-ds-0f601751-j488 64m 0% 10346Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2230Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 84m 0% 10470Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 62m 0% 6228Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1089m 6% 6992Mi 11% 13:13:08 DEBUG --- stderr --- 13:13:08 DEBUG 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096824" 13:13:44 ERROR [http_cmd]: ERROR 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 ERROR --- status code --- 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG --- status code --- 13:13:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 ERROR --- http response --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:13:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG { 13:13:44 DEBUG { 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "status": "success", 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG "data": { 13:13:44 DEBUG "data": { 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG "data": { 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG "result": [ 13:13:44 DEBUG "result": [ 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:44 DEBUG { 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:44 DEBUG { 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:44 DEBUG "metric": { 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG "metric": { 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG "metric": { 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG "pod": "ds-idrepo-1" 13:13:44 DEBUG { 13:13:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:13:44 DEBUG { 13:13:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:13:44 DEBUG { 13:13:44 DEBUG }, 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG }, 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG }, 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "value": [ 13:13:44 DEBUG "data": { 13:13:44 DEBUG "value": [ 13:13:44 DEBUG "data": { 13:13:44 DEBUG "value": [ 13:13:44 DEBUG "data": { 13:13:44 DEBUG 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG 13:13:44 DEBUG "result": [ 13:13:44 DEBUG 13:13:44 DEBUG "result": [ 13:13:44 DEBUG 13:13:44 DEBUG "result": [ 13:13:44 DEBUG 13:13:44 DEBUG { 13:13:44 DEBUG 13:13:44 DEBUG { 13:13:44 DEBUG 13:13:44 DEBUG { 13:13:44 DEBUG [...] 13:13:44 DEBUG "metric": { 13:13:44 DEBUG [...] 13:13:44 DEBUG "metric": { 13:13:44 DEBUG [...] 13:13:44 DEBUG "metric": { 13:13:44 DEBUG 13:13:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:13:44 DEBUG 13:13:44 DEBUG "instance": "10.142.0.107:9100" 13:13:44 DEBUG 13:13:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:13:44 DEBUG 13:13:44 DEBUG }, 13:13:44 DEBUG 13:13:44 DEBUG }, 13:13:44 DEBUG 13:13:44 DEBUG }, 13:13:44 DEBUG 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG "value": [ 13:13:44 DEBUG "pod": "ds-cts-1" 13:13:44 DEBUG 13:13:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:13:44 DEBUG 13:13:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:13:44 DEBUG 13:13:44 DEBUG }, 13:13:44 DEBUG 13:13:44 DEBUG }, 13:13:44 DEBUG 13:13:44 DEBUG }, 13:13:44 DEBUG 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG [...] 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG [...] 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG [...] 13:13:44 DEBUG "0" 13:13:44 DEBUG 13:13:44 DEBUG "0" 13:13:44 DEBUG 13:13:44 DEBUG "1.0469791966409463" 13:13:44 DEBUG 13:13:44 DEBUG ] 13:13:44 DEBUG 13:13:44 DEBUG ] 13:13:44 DEBUG 13:13:44 DEBUG ] 13:13:44 DEBUG 13:13:44 DEBUG } 13:13:44 DEBUG 13:13:44 DEBUG } 13:13:44 DEBUG 13:13:44 DEBUG } 13:13:44 DEBUG 13:13:44 DEBUG ] 13:13:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:13:44 DEBUG ] 13:13:44 DEBUG "instance": "10.142.0.43:9100" 13:13:44 DEBUG ] 13:13:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:13:44 DEBUG } 13:13:44 DEBUG }, 13:13:44 DEBUG } 13:13:44 DEBUG }, 13:13:44 DEBUG } 13:13:44 DEBUG }, 13:13:44 DEBUG } 13:13:44 DEBUG "value": [ 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 DEBUG } 13:13:44 DEBUG "value": [ 13:13:44 DEBUG } 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG "5412.2" 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG ] 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG ] 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "ds-idrepo-1" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "ds-idrepo-0" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "instance": "10.142.0.107:9100" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "instance": "10.142.0.43:9100" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "13020572672" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "ds-idrepo-1" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "ds-cts-1" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "instance": "10.142.0.107:9100" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "instance": "10.142.0.43:9100" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "11990.533333333331" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "ds-idrepo-1" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "ds-idrepo-0" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096824" 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "instance": "10.142.0.107:9100" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "instance": "10.142.0.43:9100" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0.0004333333333332708" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "2.0103590816083114" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "21384309.4000663" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "ds-idrepo-1" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "ds-idrepo-0" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "3622772736" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "ds-idrepo-1" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "ds-idrepo-0" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0.037" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096824" 13:13:44 INFO [http_cmd]: http status code OK 13:13:44 DEBUG --- status code --- 13:13:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:44 DEBUG --- http response --- 13:13:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:13:44 DEBUG ----- output ----- 13:13:44 DEBUG { 13:13:44 DEBUG "status": "success", 13:13:44 DEBUG "data": { 13:13:44 DEBUG "resultType": "vector", 13:13:44 DEBUG "result": [ 13:13:44 DEBUG { 13:13:44 DEBUG "metric": { 13:13:44 DEBUG "pod": "ds-idrepo-1" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG [...] 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG 13:13:44 DEBUG "pod": "ds-idrepo-0" 13:13:44 DEBUG }, 13:13:44 DEBUG "value": [ 13:13:44 DEBUG 1713096824, 13:13:44 DEBUG "0" 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG ] 13:13:44 DEBUG } 13:13:44 DEBUG } 13:13:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096825" 13:13:45 INFO [http_cmd]: http status code OK 13:13:45 DEBUG --- status code --- 13:13:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:13:45 DEBUG --- http response --- 13:13:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:13:45 DEBUG ----- output ----- 13:13:45 DEBUG { 13:13:45 DEBUG "status": "success", 13:13:45 DEBUG "data": { 13:13:45 DEBUG "resultType": "vector", 13:13:45 DEBUG "result": [ 13:13:45 DEBUG { 13:13:45 DEBUG "metric": { 13:13:45 DEBUG "instance": "10.142.0.107:9100" 13:13:45 DEBUG }, 13:13:45 DEBUG "value": [ 13:13:45 DEBUG 13:13:45 DEBUG 13:13:45 DEBUG 13:13:45 DEBUG [...] 13:13:45 DEBUG 13:13:45 DEBUG 13:13:45 DEBUG 13:13:45 DEBUG "instance": "10.142.0.43:9100" 13:13:45 DEBUG }, 13:13:45 DEBUG "value": [ 13:13:45 DEBUG 1713096825, 13:13:45 DEBUG "0.0006666666666666747" 13:13:45 DEBUG ] 13:13:45 DEBUG } 13:13:45 DEBUG ] 13:13:45 DEBUG } 13:13:45 DEBUG } 13:14:01 INFO 13:14:01 INFO [loop_until]: kubectl --namespace=xlou top pods 13:14:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:14:01 INFO [loop_until]: OK (rc = 0) 13:14:01 DEBUG --- stdout --- 13:14:01 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8531Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 7m 8892Mi ds-cts-0 15m 8090Mi ds-cts-1 16m 8376Mi ds-cts-2 15m 8104Mi ds-idrepo-0 10m 4557Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 11m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 5m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 996m 3805Mi 13:14:01 DEBUG --- stderr --- 13:14:01 DEBUG 13:14:08 INFO 13:14:08 INFO [loop_until]: kubectl --namespace=xlou top node 13:14:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:14:08 INFO [loop_until]: OK (rc = 0) 13:14:08 DEBUG --- stdout --- 13:14:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4869Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 10741Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10598Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3057Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 6591Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 12064Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2930Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 10056Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 64m 0% 5729Mi 5% gke-xlou-cdm-ds-0f601751-j488 68m 0% 10351Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2240Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 86m 0% 10471Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 6231Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 7608Mi 12% 13:14:08 DEBUG --- stderr --- 13:14:08 DEBUG 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096884" 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096884" 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 ERROR [http_cmd]: ERROR 13:14:44 ERROR --- status code --- 13:14:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:14:44 ERROR --- http response --- 13:14:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:14:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "instance": "10.142.0.107:9100" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "instance": "10.142.0.43:9100" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "7042.099999999999" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:14:44 DEBUG [...] 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG 13:14:44 DEBUG { 13:14:44 DEBUG 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG 13:14:44 DEBUG "data": { 13:14:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG }, 13:14:44 DEBUG "result": [ 13:14:44 DEBUG "value": [ 13:14:44 DEBUG { 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "0" 13:14:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:14:44 DEBUG ] 13:14:44 DEBUG }, 13:14:44 DEBUG } 13:14:44 DEBUG "value": [ 13:14:44 DEBUG ] 13:14:44 DEBUG 13:14:44 DEBUG } 13:14:44 DEBUG 13:14:44 DEBUG } 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "1.0318655773995549" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "ds-idrepo-1" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "ds-cts-1" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "ds-idrepo-1" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "ds-idrepo-0" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "ds-idrepo-1" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "ds-cts-1" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "instance": "10.142.0.107:9100" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "instance": "10.142.0.43:9100" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "13021107200" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "ds-idrepo-1" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "ds-idrepo-0" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "instance": "10.142.0.107:9100" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "instance": "10.142.0.43:9100" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "12445.666666666664" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "instance": "10.142.0.107:9100" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "instance": "10.142.0.43:9100" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0.0004666666666667633" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096884" 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "4234637312" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "ds-idrepo-0" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "ds-idrepo-0" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "ds-idrepo-1" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "ds-idrepo-0" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "ds-idrepo-1" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "ds-idrepo-0" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0.037" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096884" 13:14:44 INFO [http_cmd]: http status code OK 13:14:44 DEBUG --- status code --- 13:14:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:44 DEBUG --- http response --- 13:14:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:14:44 DEBUG ----- output ----- 13:14:44 DEBUG { 13:14:44 DEBUG "status": "success", 13:14:44 DEBUG "data": { 13:14:44 DEBUG "resultType": "vector", 13:14:44 DEBUG "result": [ 13:14:44 DEBUG { 13:14:44 DEBUG "metric": { 13:14:44 DEBUG "pod": "ds-idrepo-1" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG [...] 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG 13:14:44 DEBUG "pod": "ds-idrepo-0" 13:14:44 DEBUG }, 13:14:44 DEBUG "value": [ 13:14:44 DEBUG 1713096884, 13:14:44 DEBUG "0" 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG ] 13:14:44 DEBUG } 13:14:44 DEBUG } 13:14:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096885" 13:14:45 INFO [http_cmd]: http status code OK 13:14:45 DEBUG --- status code --- 13:14:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:14:45 DEBUG --- http response --- 13:14:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:14:45 DEBUG ----- output ----- 13:14:45 DEBUG { 13:14:45 DEBUG "status": "success", 13:14:45 DEBUG "data": { 13:14:45 DEBUG "resultType": "vector", 13:14:45 DEBUG "result": [ 13:14:45 DEBUG { 13:14:45 DEBUG "metric": { 13:14:45 DEBUG "instance": "10.142.0.107:9100" 13:14:45 DEBUG }, 13:14:45 DEBUG "value": [ 13:14:45 DEBUG 13:14:45 DEBUG 13:14:45 DEBUG 13:14:45 DEBUG [...] 13:14:45 DEBUG 13:14:45 DEBUG 13:14:45 DEBUG 13:14:45 DEBUG "instance": "10.142.0.43:9100" 13:14:45 DEBUG }, 13:14:45 DEBUG "value": [ 13:14:45 DEBUG 1713096885, 13:14:45 DEBUG "0" 13:14:45 DEBUG ] 13:14:45 DEBUG } 13:14:45 DEBUG ] 13:14:45 DEBUG } 13:14:45 DEBUG } 13:15:01 INFO 13:15:01 INFO [loop_until]: kubectl --namespace=xlou top pods 13:15:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:15:01 INFO [loop_until]: OK (rc = 0) 13:15:01 DEBUG --- stdout --- 13:15:01 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8532Mi am-7d597cf7d6-8vr8f 6m 8603Mi am-7d597cf7d6-9wz9h 7m 8892Mi ds-cts-0 17m 8090Mi ds-cts-1 15m 8375Mi ds-cts-2 19m 8104Mi ds-idrepo-0 48m 4556Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 10m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 985m 4098Mi 13:15:01 DEBUG --- stderr --- 13:15:01 DEBUG 13:15:08 INFO 13:15:08 INFO [loop_until]: kubectl --namespace=xlou top node 13:15:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:15:08 INFO [loop_until]: OK (rc = 0) 13:15:08 DEBUG --- stdout --- 13:15:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4864Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 10749Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 51m 0% 10597Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 92m 0% 3068Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 66m 0% 6592Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 12064Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2938Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 58m 0% 10061Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 5715Mi 5% gke-xlou-cdm-ds-0f601751-j488 68m 0% 10347Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2230Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 85m 0% 10475Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 102m 0% 6237Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1080m 6% 7814Mi 13% 13:15:08 DEBUG --- stderr --- 13:15:08 DEBUG 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096944" 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713096944" 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "instance": "10.142.0.107:9100" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "instance": "10.142.0.43:9100" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "5314.733333333333" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 ERROR [http_cmd]: ERROR 13:15:44 ERROR --- status code --- 13:15:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:15:44 ERROR --- http response --- 13:15:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:15:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "1.031540887312416" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-idrepo-2" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "5502.705526738067" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "ds-idrepo-1" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-cts-1" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "ds-idrepo-1" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-idrepo-0" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096944" 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "instance": "10.142.0.107:9100" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "instance": "10.142.0.43:9100" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "13021727744" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "ds-idrepo-1" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-cts-1" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "ds-idrepo-1" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-idrepo-0" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "instance": "10.142.0.107:9100" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "instance": "10.142.0.43:9100" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "12321.933333333332" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "instance": "10.142.0.107:9100" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "instance": "10.142.0.43:9100" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0.0004333333333332708" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713096944" 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "4392529920" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-idrepo-2" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-idrepo-2" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0.010380900847115035" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-idrepo-2" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "21399.410381759153" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "ds-idrepo-1" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-idrepo-0" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "ds-idrepo-1" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-idrepo-0" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0.037" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713096944" 13:15:44 INFO [http_cmd]: http status code OK 13:15:44 DEBUG --- status code --- 13:15:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:44 DEBUG --- http response --- 13:15:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:15:44 DEBUG ----- output ----- 13:15:44 DEBUG { 13:15:44 DEBUG "status": "success", 13:15:44 DEBUG "data": { 13:15:44 DEBUG "resultType": "vector", 13:15:44 DEBUG "result": [ 13:15:44 DEBUG { 13:15:44 DEBUG "metric": { 13:15:44 DEBUG "pod": "ds-idrepo-1" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG [...] 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG 13:15:44 DEBUG "pod": "ds-idrepo-0" 13:15:44 DEBUG }, 13:15:44 DEBUG "value": [ 13:15:44 DEBUG 1713096944, 13:15:44 DEBUG "0" 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG ] 13:15:44 DEBUG } 13:15:44 DEBUG } 13:15:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713096945" 13:15:45 INFO [http_cmd]: http status code OK 13:15:45 DEBUG --- status code --- 13:15:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:15:45 DEBUG --- http response --- 13:15:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:15:45 DEBUG ----- output ----- 13:15:45 DEBUG { 13:15:45 DEBUG "status": "success", 13:15:45 DEBUG "data": { 13:15:45 DEBUG "resultType": "vector", 13:15:45 DEBUG "result": [ 13:15:45 DEBUG { 13:15:45 DEBUG "metric": { 13:15:45 DEBUG "instance": "10.142.0.107:9100" 13:15:45 DEBUG }, 13:15:45 DEBUG "value": [ 13:15:45 DEBUG 13:15:45 DEBUG 13:15:45 DEBUG 13:15:45 DEBUG [...] 13:15:45 DEBUG 13:15:45 DEBUG 13:15:45 DEBUG 13:15:45 DEBUG "instance": "10.142.0.43:9100" 13:15:45 DEBUG }, 13:15:45 DEBUG "value": [ 13:15:45 DEBUG 1713096945, 13:15:45 DEBUG "0.000333333333333341" 13:15:45 DEBUG ] 13:15:45 DEBUG } 13:15:45 DEBUG ] 13:15:45 DEBUG } 13:15:45 DEBUG } 13:16:01 INFO 13:16:01 INFO [loop_until]: kubectl --namespace=xlou top pods 13:16:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:16:01 INFO [loop_until]: OK (rc = 0) 13:16:01 DEBUG --- stdout --- 13:16:01 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 4m 8532Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 6m 8892Mi ds-cts-0 18m 8059Mi ds-cts-1 12m 8376Mi ds-cts-2 17m 8077Mi ds-idrepo-0 11m 4556Mi ds-idrepo-1 10m 4316Mi ds-idrepo-2 11m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 655m 4090Mi 13:16:01 DEBUG --- stderr --- 13:16:01 DEBUG 13:16:08 INFO 13:16:08 INFO [loop_until]: kubectl --namespace=xlou top node 13:16:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:16:08 INFO [loop_until]: OK (rc = 0) 13:16:08 DEBUG --- stdout --- 13:16:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4875Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 10758Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10606Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3071Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 66m 0% 6591Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 12073Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2931Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 58m 0% 10063Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 5713Mi 5% gke-xlou-cdm-ds-0f601751-j488 62m 0% 10344Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2229Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 84m 0% 10458Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 63m 0% 6233Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 816m 5% 7849Mi 13% 13:16:08 DEBUG --- stderr --- 13:16:08 DEBUG 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 ERROR [http_cmd]: ERROR 13:16:44 ERROR --- status code --- 13:16:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 ERROR --- http response --- 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:16:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "instance": "10.142.0.107:9100" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "instance": "10.142.0.43:9100" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "5193.4" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0.6853204063759439" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG { 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:16:44 DEBUG "data": { 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG "pod": "ds-idrepo-1" 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG }, 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG "value": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:16:44 DEBUG 13:16:44 DEBUG "pod": "ds-idrepo-1" 13:16:44 DEBUG }, 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG { 13:16:44 DEBUG [...] 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "data": { 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG "result": [ 13:16:44 DEBUG "pod": "ds-cts-1" 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG { 13:16:44 DEBUG }, 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG 13:16:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:16:44 DEBUG }, 13:16:44 DEBUG "0" 13:16:44 DEBUG "pod": "ds-idrepo-0" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG ] 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG } 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG 13:16:44 DEBUG ] 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0" 13:16:44 DEBUG 13:16:44 DEBUG } 13:16:44 DEBUG "0" 13:16:44 DEBUG ] 13:16:44 DEBUG [...] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG } 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "1646.5935807551293" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "ds-idrepo-1" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "ds-cts-1" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "instance": "10.142.0.107:9100" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "instance": "10.142.0.43:9100" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "13022348288" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:16:44 DEBUG 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG 13:16:44 DEBUG { 13:16:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG }, 13:16:44 DEBUG "data": { 13:16:44 DEBUG "value": [ 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "result": [ 13:16:44 DEBUG "0" 13:16:44 DEBUG { 13:16:44 DEBUG ] 13:16:44 DEBUG "metric": { 13:16:44 DEBUG } 13:16:44 DEBUG "pod": "ds-idrepo-1" 13:16:44 DEBUG ] 13:16:44 DEBUG }, 13:16:44 DEBUG } 13:16:44 DEBUG "value": [ 13:16:44 DEBUG } 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "ds-idrepo-0" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "instance": "10.142.0.107:9100" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "instance": "10.142.0.43:9100" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "12194.166666666666" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "instance": "10.142.0.107:9100" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "instance": "10.142.0.43:9100" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0.00036666666666675953" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "4385382400" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "ds-idrepo-1" 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:16:44 DEBUG }, 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG "value": [ 13:16:44 DEBUG { 13:16:44 DEBUG 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG 13:16:44 DEBUG "data": { 13:16:44 DEBUG 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG [...] 13:16:44 DEBUG "result": [ 13:16:44 DEBUG 13:16:44 DEBUG { 13:16:44 DEBUG 13:16:44 DEBUG "metric": { 13:16:44 DEBUG 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG "pod": "ds-idrepo-0" 13:16:44 DEBUG }, 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG 13:16:44 DEBUG "0" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG 13:16:44 DEBUG ] 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG [...] 13:16:44 DEBUG } 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG 13:16:44 DEBUG ] 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG 13:16:44 DEBUG } 13:16:44 DEBUG 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:16:44 DEBUG } 13:16:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG }, 13:16:44 DEBUG { 13:16:44 DEBUG "value": [ 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "data": { 13:16:44 DEBUG "0" 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG ] 13:16:44 DEBUG "result": [ 13:16:44 DEBUG } 13:16:44 DEBUG { 13:16:44 DEBUG ] 13:16:44 DEBUG "metric": { 13:16:44 DEBUG } 13:16:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:16:44 DEBUG } 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0.0019293338521025251" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "299743288.33416086" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0.037" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "ds-idrepo-1" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "ds-idrepo-0" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097004" 13:16:44 INFO [http_cmd]: http status code OK 13:16:44 DEBUG --- status code --- 13:16:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:44 DEBUG --- http response --- 13:16:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:16:44 DEBUG ----- output ----- 13:16:44 DEBUG { 13:16:44 DEBUG "status": "success", 13:16:44 DEBUG "data": { 13:16:44 DEBUG "resultType": "vector", 13:16:44 DEBUG "result": [ 13:16:44 DEBUG { 13:16:44 DEBUG "metric": { 13:16:44 DEBUG "pod": "ds-idrepo-1" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG [...] 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG 13:16:44 DEBUG "pod": "ds-idrepo-0" 13:16:44 DEBUG }, 13:16:44 DEBUG "value": [ 13:16:44 DEBUG 1713097004, 13:16:44 DEBUG "0" 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG ] 13:16:44 DEBUG } 13:16:44 DEBUG } 13:16:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097005" 13:16:45 INFO [http_cmd]: http status code OK 13:16:45 DEBUG --- status code --- 13:16:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:16:45 DEBUG --- http response --- 13:16:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:16:45 DEBUG ----- output ----- 13:16:45 DEBUG { 13:16:45 DEBUG "status": "success", 13:16:45 DEBUG "data": { 13:16:45 DEBUG "resultType": "vector", 13:16:45 DEBUG "result": [ 13:16:45 DEBUG { 13:16:45 DEBUG "metric": { 13:16:45 DEBUG "instance": "10.142.0.107:9100" 13:16:45 DEBUG }, 13:16:45 DEBUG "value": [ 13:16:45 DEBUG 13:16:45 DEBUG 13:16:45 DEBUG 13:16:45 DEBUG [...] 13:16:45 DEBUG 13:16:45 DEBUG 13:16:45 DEBUG 13:16:45 DEBUG "instance": "10.142.0.43:9100" 13:16:45 DEBUG }, 13:16:45 DEBUG "value": [ 13:16:45 DEBUG 1713097005, 13:16:45 DEBUG "0" 13:16:45 DEBUG ] 13:16:45 DEBUG } 13:16:45 DEBUG ] 13:16:45 DEBUG } 13:16:45 DEBUG } 13:17:01 INFO 13:17:01 INFO [loop_until]: kubectl --namespace=xlou top pods 13:17:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:17:02 INFO [loop_until]: OK (rc = 0) 13:17:02 DEBUG --- stdout --- 13:17:02 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8532Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 7m 8892Mi ds-cts-0 13m 8060Mi ds-cts-1 12m 8375Mi ds-cts-2 13m 8075Mi ds-idrepo-0 11m 4555Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 10m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 4m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1001m 4467Mi 13:17:02 DEBUG --- stderr --- 13:17:02 DEBUG 13:17:08 INFO 13:17:08 INFO [loop_until]: kubectl --namespace=xlou top node 13:17:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:17:08 INFO [loop_until]: OK (rc = 0) 13:17:08 DEBUG --- stdout --- 13:17:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4874Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 10748Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10597Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 93m 0% 3068Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 65m 0% 6594Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 137m 0% 12083Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2935Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 58m 0% 10062Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 67m 0% 5733Mi 5% gke-xlou-cdm-ds-0f601751-j488 65m 0% 10340Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 68m 0% 2216Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 84m 0% 10462Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 6237Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 8325Mi 14% 13:17:08 DEBUG --- stderr --- 13:17:08 DEBUG 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "instance": "10.142.0.107:9100" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "instance": "10.142.0.43:9100" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "5313.433333333333" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713097064" 13:17:44 ERROR [http_cmd]: ERROR 13:17:44 ERROR --- status code --- 13:17:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:17:44 ERROR --- http response --- 13:17:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:17:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713097064" 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "1.034034273391076" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "ds-idrepo-1" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "ds-idrepo-0" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG { 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG "data": { 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "data": { 13:17:44 DEBUG "pod": "ds-idrepo-1" 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG }, 13:17:44 DEBUG "result": [ 13:17:44 DEBUG "value": [ 13:17:44 DEBUG { 13:17:44 DEBUG 13:17:44 DEBUG "metric": { 13:17:44 DEBUG 13:17:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:17:44 DEBUG 13:17:44 DEBUG }, 13:17:44 DEBUG [...] 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "ds-cts-1" 13:17:44 DEBUG [...] 13:17:44 DEBUG }, 13:17:44 DEBUG 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG 13:17:44 DEBUG "0" 13:17:44 DEBUG "pod": "idm-b85f9d978-rl76f" 13:17:44 DEBUG ] 13:17:44 DEBUG }, 13:17:44 DEBUG } 13:17:44 DEBUG "value": [ 13:17:44 DEBUG ] 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG } 13:17:44 DEBUG "510.48449914316876" 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "instance": "10.142.0.107:9100" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "instance": "10.142.0.43:9100" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "13023001600" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "ds-idrepo-1" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "ds-cts-1" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "ds-idrepo-1" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "ds-idrepo-0" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097064" 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "instance": "10.142.0.107:9100" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "instance": "10.142.0.43:9100" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0.0004666666666667633" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "instance": "10.142.0.107:9100" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "instance": "10.142.0.43:9100" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "12222.466666666664" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "4899692544" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG { 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "ds-idrepo-1" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "ds-idrepo-0" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "idm-b85f9d978-rl76f" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "ds-idrepo-0" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "ds-idrepo-0" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:44 INFO [http_cmd]: http status code OK 13:17:44 DEBUG --- status code --- 13:17:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:44 DEBUG --- http response --- 13:17:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:44 DEBUG ----- output ----- 13:17:44 DEBUG { 13:17:44 DEBUG "status": "success", 13:17:44 DEBUG "data": { 13:17:44 DEBUG "resultType": "vector", 13:17:44 DEBUG "result": [ 13:17:44 DEBUG { 13:17:44 DEBUG "metric": { 13:17:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG [...] 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG 13:17:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:17:44 DEBUG }, 13:17:44 DEBUG "value": [ 13:17:44 DEBUG 1713097064, 13:17:44 DEBUG "0.037" 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG ] 13:17:44 DEBUG } 13:17:44 DEBUG } 13:17:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097064" 13:17:45 INFO [http_cmd]: http status code OK 13:17:45 DEBUG --- status code --- 13:17:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:45 DEBUG --- http response --- 13:17:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:45 DEBUG ----- output ----- 13:17:45 DEBUG { 13:17:45 DEBUG "status": "success", 13:17:45 DEBUG "data": { 13:17:45 DEBUG "resultType": "vector", 13:17:45 DEBUG "result": [ 13:17:45 DEBUG { 13:17:45 DEBUG "metric": { 13:17:45 DEBUG "pod": "ds-idrepo-1" 13:17:45 DEBUG }, 13:17:45 DEBUG "value": [ 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG [...] 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG "pod": "ds-idrepo-0" 13:17:45 DEBUG }, 13:17:45 DEBUG "value": [ 13:17:45 DEBUG 1713097064, 13:17:45 DEBUG "0" 13:17:45 DEBUG ] 13:17:45 DEBUG } 13:17:45 DEBUG ] 13:17:45 DEBUG } 13:17:45 DEBUG } 13:17:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097065" 13:17:45 INFO [http_cmd]: http status code OK 13:17:45 DEBUG --- status code --- 13:17:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:45 DEBUG --- http response --- 13:17:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:17:45 DEBUG ----- output ----- 13:17:45 DEBUG { 13:17:45 DEBUG "status": "success", 13:17:45 DEBUG "data": { 13:17:45 DEBUG "resultType": "vector", 13:17:45 DEBUG "result": [ 13:17:45 DEBUG { 13:17:45 DEBUG "metric": { 13:17:45 DEBUG "pod": "ds-idrepo-1" 13:17:45 DEBUG }, 13:17:45 DEBUG "value": [ 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG [...] 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG "pod": "ds-idrepo-0" 13:17:45 DEBUG }, 13:17:45 DEBUG "value": [ 13:17:45 DEBUG 1713097065, 13:17:45 DEBUG "0" 13:17:45 DEBUG ] 13:17:45 DEBUG } 13:17:45 DEBUG ] 13:17:45 DEBUG } 13:17:45 DEBUG } 13:17:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097065" 13:17:45 INFO [http_cmd]: http status code OK 13:17:45 DEBUG --- status code --- 13:17:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:17:45 DEBUG --- http response --- 13:17:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:17:45 DEBUG ----- output ----- 13:17:45 DEBUG { 13:17:45 DEBUG "status": "success", 13:17:45 DEBUG "data": { 13:17:45 DEBUG "resultType": "vector", 13:17:45 DEBUG "result": [ 13:17:45 DEBUG { 13:17:45 DEBUG "metric": { 13:17:45 DEBUG "instance": "10.142.0.107:9100" 13:17:45 DEBUG }, 13:17:45 DEBUG "value": [ 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG [...] 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG 13:17:45 DEBUG "instance": "10.142.0.43:9100" 13:17:45 DEBUG }, 13:17:45 DEBUG "value": [ 13:17:45 DEBUG 1713097065, 13:17:45 DEBUG "0" 13:17:45 DEBUG ] 13:17:45 DEBUG } 13:17:45 DEBUG ] 13:17:45 DEBUG } 13:17:45 DEBUG } 13:18:02 INFO 13:18:02 INFO [loop_until]: kubectl --namespace=xlou top pods 13:18:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:18:02 INFO [loop_until]: OK (rc = 0) 13:18:02 DEBUG --- stdout --- 13:18:02 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8532Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 6m 8892Mi ds-cts-0 13m 8059Mi ds-cts-1 10m 8376Mi ds-cts-2 12m 8076Mi ds-idrepo-0 10m 4556Mi ds-idrepo-1 11m 4317Mi ds-idrepo-2 11m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 3m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 71Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 999m 4861Mi 13:18:02 DEBUG --- stderr --- 13:18:02 DEBUG 13:18:08 INFO 13:18:08 INFO [loop_until]: kubectl --namespace=xlou top node 13:18:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:18:08 INFO [loop_until]: OK (rc = 0) 13:18:08 DEBUG --- stdout --- 13:18:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4866Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 10739Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10595Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3061Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 64m 0% 6601Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 12068Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2943Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 60m 0% 10055Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 61m 0% 5718Mi 5% gke-xlou-cdm-ds-0f601751-j488 62m 0% 10350Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2220Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 74m 0% 10463Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 6231Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 8639Mi 14% 13:18:08 DEBUG --- stderr --- 13:18:08 DEBUG 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 ERROR [http_cmd]: ERROR 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 ERROR --- status code --- 13:18:44 DEBUG --- status code --- 13:18:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 ERROR --- http response --- 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:18:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:18:44 DEBUG { 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG { 13:18:44 DEBUG "data": { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "result": [ 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG { 13:18:44 DEBUG "result": [ 13:18:44 DEBUG "metric": { 13:18:44 DEBUG { 13:18:44 DEBUG "instance": "10.142.0.107:9100" 13:18:44 DEBUG "metric": { 13:18:44 DEBUG }, 13:18:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:18:44 DEBUG "value": [ 13:18:44 DEBUG }, 13:18:44 DEBUG 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "instance": "10.142.0.43:9100" 13:18:44 DEBUG 13:18:44 DEBUG }, 13:18:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:18:44 DEBUG "value": [ 13:18:44 DEBUG }, 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG "5467.966666666666" 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG ] 13:18:44 DEBUG "1.033000662401988" 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "ds-idrepo-1" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "ds-idrepo-0" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097124" 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "ds-idrepo-1" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "ds-cts-1" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "instance": "10.142.0.107:9100" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "instance": "10.142.0.43:9100" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "13023736832" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "ds-idrepo-1" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "ds-cts-1" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "ds-idrepo-1" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "ds-idrepo-0" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097124" 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "instance": "10.142.0.107:9100" 13:18:44 DEBUG "instance": "10.142.0.107:9100" 13:18:44 DEBUG }, 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "instance": "10.142.0.43:9100" 13:18:44 DEBUG "instance": "10.142.0.43:9100" 13:18:44 DEBUG }, 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0.000333333333333267" 13:18:44 DEBUG "12360.799999999997" 13:18:44 DEBUG ] 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "5195317248" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097124" 13:18:44 INFO [http_cmd]: http status code OK 13:18:44 DEBUG --- status code --- 13:18:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:44 DEBUG --- http response --- 13:18:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:44 DEBUG ----- output ----- 13:18:44 DEBUG { 13:18:44 DEBUG "status": "success", 13:18:44 DEBUG "data": { 13:18:44 DEBUG "resultType": "vector", 13:18:44 DEBUG "result": [ 13:18:44 DEBUG { 13:18:44 DEBUG "metric": { 13:18:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG [...] 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG 13:18:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:18:44 DEBUG }, 13:18:44 DEBUG "value": [ 13:18:44 DEBUG 1713097124, 13:18:44 DEBUG "0" 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG ] 13:18:44 DEBUG } 13:18:44 DEBUG } 13:18:45 INFO [http_cmd]: http status code OK 13:18:45 DEBUG --- status code --- 13:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:45 DEBUG --- http response --- 13:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:45 DEBUG ----- output ----- 13:18:45 DEBUG { 13:18:45 DEBUG "status": "success", 13:18:45 DEBUG "data": { 13:18:45 DEBUG "resultType": "vector", 13:18:45 DEBUG "result": [ 13:18:45 DEBUG { 13:18:45 DEBUG "metric": { 13:18:45 DEBUG "pod": "ds-idrepo-1" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG [...] 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG "pod": "ds-idrepo-0" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 INFO [http_cmd]: http status code OK 13:18:45 DEBUG 1713097124, 13:18:45 DEBUG --- status code --- 13:18:45 DEBUG "0" 13:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:45 DEBUG ] 13:18:45 DEBUG --- http response --- 13:18:45 DEBUG } 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG } 13:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:18:45 DEBUG ----- output ----- 13:18:45 DEBUG { 13:18:45 DEBUG "status": "success", 13:18:45 DEBUG "data": { 13:18:45 DEBUG "resultType": "vector", 13:18:45 DEBUG "result": [ 13:18:45 DEBUG { 13:18:45 DEBUG "metric": { 13:18:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG [...] 13:18:45 DEBUG 13:18:45 INFO [http_cmd]: http status code OK 13:18:45 DEBUG 13:18:45 DEBUG --- status code --- 13:18:45 DEBUG 13:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:45 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 13:18:45 DEBUG --- http response --- 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 1713097124, 13:18:45 DEBUG "0" 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG } 13:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:18:45 DEBUG ----- output ----- 13:18:45 DEBUG { 13:18:45 DEBUG "status": "success", 13:18:45 DEBUG "data": { 13:18:45 DEBUG "resultType": "vector", 13:18:45 DEBUG "result": [ 13:18:45 DEBUG { 13:18:45 DEBUG "metric": { 13:18:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG [...] 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG "pod": "ds-idrepo-0" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 1713097124, 13:18:45 DEBUG "0" 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG } 13:18:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097125" 13:18:45 INFO [http_cmd]: http status code OK 13:18:45 DEBUG --- status code --- 13:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:45 DEBUG --- http response --- 13:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:18:45 DEBUG ----- output ----- 13:18:45 DEBUG { 13:18:45 DEBUG "status": "success", 13:18:45 DEBUG "data": { 13:18:45 DEBUG "resultType": "vector", 13:18:45 DEBUG "result": [ 13:18:45 DEBUG { 13:18:45 DEBUG "metric": { 13:18:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG [...] 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG "pod": "ds-idrepo-0" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 1713097125, 13:18:45 DEBUG "0" 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG } 13:18:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713097125" 13:18:45 INFO [http_cmd]: http status code OK 13:18:45 DEBUG --- status code --- 13:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:45 DEBUG --- http response --- 13:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:45 DEBUG ----- output ----- 13:18:45 DEBUG { 13:18:45 DEBUG "status": "success", 13:18:45 DEBUG "data": { 13:18:45 DEBUG "resultType": "vector", 13:18:45 DEBUG "result": [ 13:18:45 DEBUG { 13:18:45 DEBUG "metric": { 13:18:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG [...] 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 1713097125, 13:18:45 DEBUG "0.037" 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG } 13:18:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097125" 13:18:45 INFO [http_cmd]: http status code OK 13:18:45 DEBUG --- status code --- 13:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:45 DEBUG --- http response --- 13:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:45 DEBUG ----- output ----- 13:18:45 DEBUG { 13:18:45 DEBUG "status": "success", 13:18:45 DEBUG "data": { 13:18:45 DEBUG "resultType": "vector", 13:18:45 DEBUG "result": [ 13:18:45 DEBUG { 13:18:45 DEBUG "metric": { 13:18:45 DEBUG "pod": "ds-idrepo-1" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG [...] 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG "pod": "ds-idrepo-0" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 1713097125, 13:18:45 DEBUG "0" 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG } 13:18:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097125" 13:18:45 INFO [http_cmd]: http status code OK 13:18:45 DEBUG --- status code --- 13:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:45 DEBUG --- http response --- 13:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:18:45 DEBUG ----- output ----- 13:18:45 DEBUG { 13:18:45 DEBUG "status": "success", 13:18:45 DEBUG "data": { 13:18:45 DEBUG "resultType": "vector", 13:18:45 DEBUG "result": [ 13:18:45 DEBUG { 13:18:45 DEBUG "metric": { 13:18:45 DEBUG "pod": "ds-idrepo-1" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG [...] 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG "pod": "ds-idrepo-0" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 1713097125, 13:18:45 DEBUG "0" 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG } 13:18:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097125" 13:18:45 INFO [http_cmd]: http status code OK 13:18:45 DEBUG --- status code --- 13:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:18:45 DEBUG --- http response --- 13:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:18:45 DEBUG ----- output ----- 13:18:45 DEBUG { 13:18:45 DEBUG "status": "success", 13:18:45 DEBUG "data": { 13:18:45 DEBUG "resultType": "vector", 13:18:45 DEBUG "result": [ 13:18:45 DEBUG { 13:18:45 DEBUG "metric": { 13:18:45 DEBUG "instance": "10.142.0.107:9100" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG [...] 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG 13:18:45 DEBUG "instance": "10.142.0.43:9100" 13:18:45 DEBUG }, 13:18:45 DEBUG "value": [ 13:18:45 DEBUG 1713097125, 13:18:45 DEBUG "0.0003333333333333262" 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG ] 13:18:45 DEBUG } 13:18:45 DEBUG } 13:19:02 INFO 13:19:02 INFO [loop_until]: kubectl --namespace=xlou top pods 13:19:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:19:02 INFO [loop_until]: OK (rc = 0) 13:19:02 DEBUG --- stdout --- 13:19:02 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8531Mi am-7d597cf7d6-8vr8f 7m 8603Mi am-7d597cf7d6-9wz9h 6m 8893Mi ds-cts-0 12m 8059Mi ds-cts-1 11m 8376Mi ds-cts-2 12m 8076Mi ds-idrepo-0 11m 4556Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 10m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 5m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 990m 4878Mi 13:19:02 DEBUG --- stderr --- 13:19:02 DEBUG 13:19:08 INFO 13:19:08 INFO [loop_until]: kubectl --namespace=xlou top node 13:19:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:19:08 INFO [loop_until]: OK (rc = 0) 13:19:08 DEBUG --- stdout --- 13:19:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4874Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 10742Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 10595Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3064Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 6595Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 12083Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2936Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 57m 0% 10045Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5731Mi 5% gke-xlou-cdm-ds-0f601751-j488 64m 0% 10350Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2226Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 88m 0% 10462Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 65m 0% 6231Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 8663Mi 14% 13:19:08 DEBUG --- stderr --- 13:19:08 DEBUG 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713097184" 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097184" 13:19:44 ERROR [http_cmd]: ERROR 13:19:44 ERROR --- status code --- 13:19:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:19:44 ERROR --- http response --- 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:19:44 DEBUG --- status code --- 13:19:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "instance": "10.142.0.107:9100" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "instance": "10.142.0.43:9100" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "6958.5" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713097184" 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097184" 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097184" 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097184" 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097184" 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG { 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG "data": { 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG "result": [ 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG { 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG }, 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG "value": [ 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:19:44 DEBUG 13:19:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:19:44 DEBUG }, 13:19:44 DEBUG [...] 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG }, 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "1.0321055479613244" 13:19:44 DEBUG 13:19:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:19:44 DEBUG ] 13:19:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:19:44 DEBUG }, 13:19:44 DEBUG } 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG ] 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG } 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "0" 13:19:44 DEBUG } 13:19:44 DEBUG "0" 13:19:44 DEBUG ] 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "pod": "ds-idrepo-1" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "pod": "ds-idrepo-0" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "0" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097184" 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097184" 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "0" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097184" 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097184" 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "pod": "ds-idrepo-0" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "2791.705985063637" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "pod": "ds-idrepo-1" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "pod": "ds-cts-1" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "0" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "instance": "10.142.0.107:9100" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "instance": "10.142.0.43:9100" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "13024228352" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713097184" 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "pod": "ds-idrepo-1" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "pod": "ds-cts-1" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "0" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097184" 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "pod": "ds-idrepo-1" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "pod": "ds-idrepo-0" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "0" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097184" 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "0" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097184" 13:19:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097184" 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "instance": "10.142.0.107:9100" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "instance": "10.142.0.43:9100" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "0.00036666666666675953" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:44 INFO [http_cmd]: http status code OK 13:19:44 DEBUG --- status code --- 13:19:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:44 DEBUG --- http response --- 13:19:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:19:44 DEBUG ----- output ----- 13:19:44 DEBUG { 13:19:44 DEBUG "status": "success", 13:19:44 DEBUG "data": { 13:19:44 DEBUG "resultType": "vector", 13:19:44 DEBUG "result": [ 13:19:44 DEBUG { 13:19:44 DEBUG "metric": { 13:19:44 DEBUG "instance": "10.142.0.107:9100" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG [...] 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG 13:19:44 DEBUG "instance": "10.142.0.43:9100" 13:19:44 DEBUG }, 13:19:44 DEBUG "value": [ 13:19:44 DEBUG 1713097184, 13:19:44 DEBUG "12188.83333333333" 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG ] 13:19:44 DEBUG } 13:19:44 DEBUG } 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097185" 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG { 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG "data": { 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG "result": [ 13:19:45 DEBUG { 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG [...] 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG "pod": "ds-idrepo-0" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG "18380.00833070369" 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713097185" 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097185" 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097185" 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097185" 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG { 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG "data": { 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG "result": [ 13:19:45 DEBUG { 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG [...] 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG "0" 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG { 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG "data": { 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG "result": [ 13:19:45 DEBUG { 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG [...] 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG "5210677248" 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097185" 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097185" 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG { 13:19:45 DEBUG { 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG "data": { 13:19:45 DEBUG "data": { 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG "result": [ 13:19:45 DEBUG "result": [ 13:19:45 DEBUG { 13:19:45 DEBUG { 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:19:45 DEBUG "pod": "ds-idrepo-1" 13:19:45 DEBUG }, 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG [...] 13:19:45 DEBUG [...] 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:19:45 DEBUG "pod": "ds-idrepo-0" 13:19:45 DEBUG }, 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG "0" 13:19:45 DEBUG "0" 13:19:45 DEBUG ] 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 DEBUG ] 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG { 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG "data": { 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG "result": [ 13:19:45 DEBUG { 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 13:19:45 DEBUG 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG 13:19:45 DEBUG { 13:19:45 DEBUG [...] 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG 13:19:45 DEBUG "data": { 13:19:45 DEBUG 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG 13:19:45 DEBUG "result": [ 13:19:45 DEBUG "pod": "ds-idrepo-0" 13:19:45 DEBUG { 13:19:45 DEBUG }, 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "value": [ 13:19:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG }, 13:19:45 DEBUG "0" 13:19:45 DEBUG "value": [ 13:19:45 DEBUG ] 13:19:45 DEBUG 13:19:45 DEBUG } 13:19:45 DEBUG 13:19:45 DEBUG ] 13:19:45 DEBUG 13:19:45 DEBUG } 13:19:45 DEBUG [...] 13:19:45 DEBUG } 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG "pod": "ds-idrepo-0" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG "0.004443662661483319" 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713097185" 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG { 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG "data": { 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG "result": [ 13:19:45 DEBUG { 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG [...] 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG "0.037" 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097185" 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG { 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG "data": { 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG "result": [ 13:19:45 DEBUG { 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "pod": "ds-idrepo-1" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG [...] 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG "pod": "ds-idrepo-0" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG "0" 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097185" 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG { 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG "data": { 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG "result": [ 13:19:45 DEBUG { 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "pod": "ds-idrepo-1" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG [...] 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG "pod": "ds-idrepo-0" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG "0" 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG } 13:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097185" 13:19:45 INFO [http_cmd]: http status code OK 13:19:45 DEBUG --- status code --- 13:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:19:45 DEBUG --- http response --- 13:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:19:45 DEBUG ----- output ----- 13:19:45 DEBUG { 13:19:45 DEBUG "status": "success", 13:19:45 DEBUG "data": { 13:19:45 DEBUG "resultType": "vector", 13:19:45 DEBUG "result": [ 13:19:45 DEBUG { 13:19:45 DEBUG "metric": { 13:19:45 DEBUG "instance": "10.142.0.107:9100" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG [...] 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG 13:19:45 DEBUG "instance": "10.142.0.43:9100" 13:19:45 DEBUG }, 13:19:45 DEBUG "value": [ 13:19:45 DEBUG 1713097185, 13:19:45 DEBUG "0" 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG ] 13:19:45 DEBUG } 13:19:45 DEBUG } 13:20:02 INFO 13:20:02 INFO [loop_until]: kubectl --namespace=xlou top pods 13:20:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:20:02 INFO [loop_until]: OK (rc = 0) 13:20:02 DEBUG --- stdout --- 13:20:02 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7m 8532Mi am-7d597cf7d6-8vr8f 11m 8603Mi am-7d597cf7d6-9wz9h 7m 8892Mi ds-cts-0 12m 8059Mi ds-cts-1 12m 8375Mi ds-cts-2 13m 8077Mi ds-idrepo-0 11m 4556Mi ds-idrepo-1 10m 4318Mi ds-idrepo-2 10m 4106Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1012Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 700m 4723Mi 13:20:02 DEBUG --- stderr --- 13:20:02 DEBUG 13:20:08 INFO 13:20:08 INFO [loop_until]: kubectl --namespace=xlou top node 13:20:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:20:08 INFO [loop_until]: OK (rc = 0) 13:20:08 DEBUG --- stdout --- 13:20:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4869Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 10747Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 10594Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3050Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 65m 0% 6593Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 136m 0% 12087Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2936Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 10041Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 5735Mi 5% gke-xlou-cdm-ds-0f601751-j488 66m 0% 10350Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2226Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 80m 0% 10469Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 6223Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 491m 3% 8506Mi 14% 13:20:08 DEBUG --- stderr --- 13:20:08 DEBUG 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097244" 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:20:44 DEBUG ----- output ----- 13:20:44 DEBUG { 13:20:44 DEBUG "status": "success", 13:20:44 DEBUG "data": { 13:20:44 DEBUG "resultType": "vector", 13:20:44 DEBUG "result": [ 13:20:44 DEBUG { 13:20:44 DEBUG "metric": { 13:20:44 DEBUG "instance": "10.142.0.107:9100" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG [...] 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG "instance": "10.142.0.43:9100" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 1713097244, 13:20:44 DEBUG "7200" 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713097244" 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097244" 13:20:44 ERROR [http_cmd]: ERROR 13:20:44 ERROR --- status code --- 13:20:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:20:44 ERROR --- http response --- 13:20:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:20:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713097244" 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097244" 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097244" 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097244" 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:20:44 DEBUG ----- output ----- 13:20:44 DEBUG { 13:20:44 DEBUG "status": "success", 13:20:44 DEBUG "data": { 13:20:44 DEBUG "resultType": "vector", 13:20:44 DEBUG "result": [ 13:20:44 DEBUG { 13:20:44 DEBUG "metric": { 13:20:44 DEBUG "pod": "ds-idrepo-1" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG [...] 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG "pod": "ds-idrepo-0" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 1713097244, 13:20:44 DEBUG "0" 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:20:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:44 DEBUG ----- output ----- 13:20:44 DEBUG ----- output ----- 13:20:44 DEBUG { 13:20:44 DEBUG { 13:20:44 DEBUG "status": "success", 13:20:44 DEBUG "status": "success", 13:20:44 DEBUG "data": { 13:20:44 DEBUG "data": { 13:20:44 DEBUG "resultType": "vector", 13:20:44 DEBUG "resultType": "vector", 13:20:44 DEBUG "result": [ 13:20:44 DEBUG "result": [ 13:20:44 DEBUG { 13:20:44 DEBUG { 13:20:44 DEBUG "metric": { 13:20:44 DEBUG "metric": { 13:20:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:20:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:20:44 DEBUG }, 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG [...] 13:20:44 DEBUG [...] 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:20:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:20:44 DEBUG }, 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 1713097244, 13:20:44 DEBUG 1713097244, 13:20:44 DEBUG "0" 13:20:44 DEBUG "0" 13:20:44 DEBUG ] 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 DEBUG ] 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:20:44 DEBUG ----- output ----- 13:20:44 DEBUG { 13:20:44 DEBUG "status": "success", 13:20:44 DEBUG "data": { 13:20:44 DEBUG "resultType": "vector", 13:20:44 DEBUG "result": [ 13:20:44 DEBUG { 13:20:44 DEBUG "metric": { 13:20:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG [...] 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 1713097244, 13:20:44 DEBUG "0.5668425444395901" 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097244" 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:20:44 DEBUG ----- output ----- 13:20:44 DEBUG { 13:20:44 DEBUG "status": "success", 13:20:44 DEBUG "data": { 13:20:44 DEBUG "resultType": "vector", 13:20:44 DEBUG "result": [ 13:20:44 DEBUG { 13:20:44 DEBUG "metric": { 13:20:44 DEBUG "instance": "10.142.0.107:9100" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG [...] 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG "instance": "10.142.0.43:9100" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 1713097244, 13:20:44 DEBUG "13024840704" 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097244" 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:44 DEBUG ----- output ----- 13:20:44 DEBUG { 13:20:44 DEBUG "status": "success", 13:20:44 DEBUG "data": { 13:20:44 DEBUG "resultType": "vector", 13:20:44 DEBUG "result": [ 13:20:44 DEBUG { 13:20:44 DEBUG "metric": { 13:20:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG [...] 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 1713097244, 13:20:44 DEBUG "0" 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097244" 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097244" 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:20:44 DEBUG ----- output ----- 13:20:44 DEBUG { 13:20:44 DEBUG "status": "success", 13:20:44 DEBUG "data": { 13:20:44 DEBUG "resultType": "vector", 13:20:44 DEBUG "result": [ 13:20:44 DEBUG { 13:20:44 DEBUG "metric": { 13:20:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG [...] 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 1713097244, 13:20:44 DEBUG "769.2981906968007" 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:20:44 DEBUG ----- output ----- 13:20:44 DEBUG { 13:20:44 DEBUG "status": "success", 13:20:44 DEBUG "data": { 13:20:44 DEBUG "resultType": "vector", 13:20:44 DEBUG "result": [ 13:20:44 DEBUG { 13:20:44 DEBUG "metric": { 13:20:44 DEBUG "pod": "ds-idrepo-1" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG [...] 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG "pod": "ds-cts-1" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 1713097244, 13:20:44 DEBUG "0" 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713097244" 13:20:44 INFO [http_cmd]: http status code OK 13:20:44 DEBUG --- status code --- 13:20:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:44 DEBUG --- http response --- 13:20:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:20:44 DEBUG ----- output ----- 13:20:44 DEBUG { 13:20:44 DEBUG "status": "success", 13:20:44 DEBUG "data": { 13:20:44 DEBUG "resultType": "vector", 13:20:44 DEBUG "result": [ 13:20:44 DEBUG { 13:20:44 DEBUG "metric": { 13:20:44 DEBUG "pod": "ds-idrepo-1" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG [...] 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG 13:20:44 DEBUG "pod": "ds-cts-1" 13:20:44 DEBUG }, 13:20:44 DEBUG "value": [ 13:20:44 DEBUG 1713097244, 13:20:44 DEBUG "0" 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG ] 13:20:44 DEBUG } 13:20:44 DEBUG } 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097244" 13:20:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097244" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "instance": "10.142.0.107:9100" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "instance": "10.142.0.43:9100" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097244, 13:20:45 DEBUG "0.00040000000000001514" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "ds-idrepo-1" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "ds-idrepo-0" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097244, 13:20:45 DEBUG "0" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097245" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097245" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "instance": "10.142.0.107:9100" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "instance": "10.142.0.43:9100" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "13916.499999999996" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097245" 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097245" 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097245" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "ds-idrepo-0" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713097245" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097245" 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097245" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "5152661504" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097245" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "ds-idrepo-1" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "ds-idrepo-0" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "ds-idrepo-0" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713097245" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0.037" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097245" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "ds-idrepo-1" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "ds-idrepo-0" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097245" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "pod": "ds-idrepo-1" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "pod": "ds-idrepo-0" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097245" 13:20:45 INFO [http_cmd]: http status code OK 13:20:45 DEBUG --- status code --- 13:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:20:45 DEBUG --- http response --- 13:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:20:45 DEBUG ----- output ----- 13:20:45 DEBUG { 13:20:45 DEBUG "status": "success", 13:20:45 DEBUG "data": { 13:20:45 DEBUG "resultType": "vector", 13:20:45 DEBUG "result": [ 13:20:45 DEBUG { 13:20:45 DEBUG "metric": { 13:20:45 DEBUG "instance": "10.142.0.107:9100" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG [...] 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG 13:20:45 DEBUG "instance": "10.142.0.43:9100" 13:20:45 DEBUG }, 13:20:45 DEBUG "value": [ 13:20:45 DEBUG 1713097245, 13:20:45 DEBUG "0.0003333333333333262" 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG ] 13:20:45 DEBUG } 13:20:45 DEBUG } 13:21:02 INFO 13:21:02 INFO [loop_until]: kubectl --namespace=xlou top pods 13:21:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:21:02 INFO [loop_until]: OK (rc = 0) 13:21:02 DEBUG --- stdout --- 13:21:02 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8532Mi am-7d597cf7d6-8vr8f 6m 8603Mi am-7d597cf7d6-9wz9h 7m 8892Mi ds-cts-0 12m 8059Mi ds-cts-1 12m 8375Mi ds-cts-2 12m 8075Mi ds-idrepo-0 11m 4556Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 11m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 12m 1014Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 4m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1041m 5076Mi 13:21:02 DEBUG --- stderr --- 13:21:02 DEBUG 13:21:08 INFO 13:21:08 INFO [loop_until]: kubectl --namespace=xlou top node 13:21:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:21:08 INFO [loop_until]: OK (rc = 0) 13:21:08 DEBUG --- stdout --- 13:21:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4874Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 10747Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10591Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3047Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 6600Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 12066Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 2942Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 60m 0% 10043Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 5739Mi 5% gke-xlou-cdm-ds-0f601751-j488 64m 0% 10359Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2229Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 80m 0% 10456Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 69m 0% 6241Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1071m 6% 8896Mi 15% 13:21:08 DEBUG --- stderr --- 13:21:08 DEBUG 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097304" 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713097304" 13:21:44 INFO [http_cmd]: http status code OK 13:21:44 DEBUG --- status code --- 13:21:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:44 DEBUG --- http response --- 13:21:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:21:44 DEBUG ----- output ----- 13:21:44 DEBUG { 13:21:44 DEBUG "status": "success", 13:21:44 DEBUG "data": { 13:21:44 DEBUG "resultType": "vector", 13:21:44 DEBUG "result": [ 13:21:44 DEBUG { 13:21:44 DEBUG "metric": { 13:21:44 DEBUG "instance": "10.142.0.107:9100" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG [...] 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG "instance": "10.142.0.43:9100" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 1713097304, 13:21:44 DEBUG "7213.066666666667" 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG } 13:21:44 ERROR [http_cmd]: ERROR 13:21:44 ERROR --- status code --- 13:21:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:21:44 ERROR --- http response --- 13:21:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:21:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097304" 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097304" 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713097304" 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097304" 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097304" 13:21:44 INFO [http_cmd]: http status code OK 13:21:44 DEBUG --- status code --- 13:21:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:44 DEBUG --- http response --- 13:21:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:21:44 DEBUG ----- output ----- 13:21:44 DEBUG { 13:21:44 DEBUG "status": "success", 13:21:44 DEBUG "data": { 13:21:44 DEBUG "resultType": "vector", 13:21:44 DEBUG "result": [ 13:21:44 DEBUG { 13:21:44 DEBUG "metric": { 13:21:44 DEBUG "pod": "ds-idrepo-1" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG [...] 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG "pod": "ds-idrepo-0" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 1713097304, 13:21:44 DEBUG "0" 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG } 13:21:44 INFO [http_cmd]: http status code OK 13:21:44 DEBUG --- status code --- 13:21:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:44 DEBUG --- http response --- 13:21:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:44 DEBUG ----- output ----- 13:21:44 DEBUG { 13:21:44 DEBUG "status": "success", 13:21:44 DEBUG "data": { 13:21:44 DEBUG "resultType": "vector", 13:21:44 DEBUG "result": [ 13:21:44 DEBUG { 13:21:44 DEBUG "metric": { 13:21:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG [...] 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 1713097304, 13:21:44 DEBUG "0" 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG } 13:21:44 INFO [http_cmd]: http status code OK 13:21:44 DEBUG --- status code --- 13:21:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:44 DEBUG --- http response --- 13:21:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:21:44 INFO [http_cmd]: http status code OK 13:21:44 DEBUG --- status code --- 13:21:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:44 DEBUG --- http response --- 13:21:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:21:44 DEBUG ----- output ----- 13:21:44 DEBUG { 13:21:44 DEBUG "status": "success", 13:21:44 DEBUG "data": { 13:21:44 DEBUG "resultType": "vector", 13:21:44 DEBUG "result": [ 13:21:44 DEBUG { 13:21:44 DEBUG "metric": { 13:21:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG [...] 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 1713097304, 13:21:44 DEBUG "1.0836078691179363" 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG } 13:21:44 INFO [http_cmd]: http status code OK 13:21:44 DEBUG --- status code --- 13:21:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:44 DEBUG --- http response --- 13:21:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:44 DEBUG ----- output ----- 13:21:44 DEBUG { 13:21:44 DEBUG "status": "success", 13:21:44 DEBUG "data": { 13:21:44 DEBUG "resultType": "vector", 13:21:44 DEBUG "result": [ 13:21:44 DEBUG { 13:21:44 DEBUG "metric": { 13:21:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG [...] 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 1713097304, 13:21:44 DEBUG "0" 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG } 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097304" 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097304" 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097304" 13:21:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097304" 13:21:44 INFO [http_cmd]: http status code OK 13:21:44 DEBUG --- status code --- 13:21:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:44 DEBUG --- http response --- 13:21:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:44 DEBUG ----- output ----- 13:21:44 DEBUG { 13:21:44 DEBUG "status": "success", 13:21:44 DEBUG "data": { 13:21:44 DEBUG "resultType": "vector", 13:21:44 DEBUG "result": [ 13:21:44 DEBUG { 13:21:44 DEBUG "metric": { 13:21:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG [...] 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 1713097304, 13:21:44 DEBUG "0" 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG } 13:21:44 INFO [http_cmd]: http status code OK 13:21:44 DEBUG --- status code --- 13:21:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:44 DEBUG --- http response --- 13:21:44 INFO [http_cmd]: http status code OK 13:21:44 INFO [http_cmd]: http status code OK 13:21:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:21:44 DEBUG --- status code --- 13:21:44 DEBUG --- status code --- 13:21:44 DEBUG ----- output ----- 13:21:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:44 DEBUG { 13:21:44 DEBUG --- http response --- 13:21:44 DEBUG --- http response --- 13:21:44 DEBUG "status": "success", 13:21:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:21:44 DEBUG "data": { 13:21:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:21:44 DEBUG ----- output ----- 13:21:44 DEBUG "resultType": "vector", 13:21:44 DEBUG ----- output ----- 13:21:44 DEBUG { 13:21:44 DEBUG "result": [ 13:21:44 DEBUG { 13:21:44 DEBUG "status": "success", 13:21:44 DEBUG { 13:21:44 DEBUG "status": "success", 13:21:44 DEBUG "data": { 13:21:44 DEBUG "metric": { 13:21:44 DEBUG "data": { 13:21:44 DEBUG "resultType": "vector", 13:21:44 DEBUG "pod": "ds-idrepo-1" 13:21:44 DEBUG "resultType": "vector", 13:21:44 DEBUG "result": [ 13:21:44 DEBUG }, 13:21:44 DEBUG "result": [ 13:21:44 DEBUG { 13:21:44 DEBUG "value": [ 13:21:44 DEBUG { 13:21:44 DEBUG "metric": { 13:21:44 DEBUG 13:21:44 DEBUG "metric": { 13:21:44 DEBUG "instance": "10.142.0.107:9100" 13:21:44 DEBUG 13:21:44 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:21:44 DEBUG }, 13:21:44 DEBUG 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG [...] 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG [...] 13:21:44 DEBUG "pod": "ds-cts-1" 13:21:44 DEBUG [...] 13:21:44 DEBUG 13:21:44 DEBUG }, 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 13:21:44 DEBUG 13:21:44 DEBUG 1713097304, 13:21:44 DEBUG 13:21:44 DEBUG "instance": "10.142.0.43:9100" 13:21:44 DEBUG "0" 13:21:44 DEBUG "pod": "ds-idrepo-0" 13:21:44 DEBUG }, 13:21:44 DEBUG ] 13:21:44 DEBUG }, 13:21:44 DEBUG "value": [ 13:21:44 DEBUG } 13:21:44 DEBUG "value": [ 13:21:44 DEBUG 1713097304, 13:21:44 DEBUG ] 13:21:44 DEBUG 1713097304, 13:21:44 DEBUG "13025612800" 13:21:44 DEBUG } 13:21:44 DEBUG "5372.5078698845755" 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG } 13:21:44 DEBUG ] 13:21:44 DEBUG ] 13:21:44 DEBUG } 13:21:44 DEBUG } 13:21:44 DEBUG } 13:21:44 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "instance": "10.142.0.107:9100" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "instance": "10.142.0.43:9100" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0.00040000000000001514" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "ds-idrepo-1" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "ds-cts-1" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "ds-idrepo-1" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "ds-idrepo-0" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "instance": "10.142.0.107:9100" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "instance": "10.142.0.43:9100" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "13808.599999999997" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "ds-idrepo-0" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "ds-idrepo-0" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "27534.10283315845" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "5418561536" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "ds-idrepo-1" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "ds-idrepo-0" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "ds-idrepo-0" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0.010152282266826164" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0.037" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "ds-idrepo-1" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "ds-idrepo-0" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "pod": "ds-idrepo-1" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "pod": "ds-idrepo-0" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097305" 13:21:45 INFO [http_cmd]: http status code OK 13:21:45 DEBUG --- status code --- 13:21:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:21:45 DEBUG --- http response --- 13:21:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:21:45 DEBUG ----- output ----- 13:21:45 DEBUG { 13:21:45 DEBUG "status": "success", 13:21:45 DEBUG "data": { 13:21:45 DEBUG "resultType": "vector", 13:21:45 DEBUG "result": [ 13:21:45 DEBUG { 13:21:45 DEBUG "metric": { 13:21:45 DEBUG "instance": "10.142.0.107:9100" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG [...] 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG 13:21:45 DEBUG "instance": "10.142.0.43:9100" 13:21:45 DEBUG }, 13:21:45 DEBUG "value": [ 13:21:45 DEBUG 1713097305, 13:21:45 DEBUG "0" 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG ] 13:21:45 DEBUG } 13:21:45 DEBUG } 13:22:02 INFO 13:22:02 INFO [loop_until]: kubectl --namespace=xlou top pods 13:22:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:22:02 INFO [loop_until]: OK (rc = 0) 13:22:02 DEBUG --- stdout --- 13:22:02 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8532Mi am-7d597cf7d6-8vr8f 8m 8603Mi am-7d597cf7d6-9wz9h 7m 8892Mi ds-cts-0 11m 8059Mi ds-cts-1 11m 8376Mi ds-cts-2 12m 8075Mi ds-idrepo-0 11m 4556Mi ds-idrepo-1 11m 4317Mi ds-idrepo-2 11m 4107Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1014Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 4m 70Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1000m 5081Mi 13:22:02 DEBUG --- stderr --- 13:22:02 DEBUG 13:22:09 INFO 13:22:09 INFO [loop_until]: kubectl --namespace=xlou top node 13:22:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:22:09 INFO [loop_until]: OK (rc = 0) 13:22:09 DEBUG --- stdout --- 13:22:09 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4878Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 10746Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10590Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3051Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 66m 0% 6612Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 12072Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2946Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 10058Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 60m 0% 5732Mi 5% gke-xlou-cdm-ds-0f601751-j488 63m 0% 10352Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2228Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 82m 0% 10460Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 6232Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 8898Mi 15% 13:22:09 DEBUG --- stderr --- 13:22:09 DEBUG 13:22:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097364" 13:22:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713097364" 13:22:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097364" 13:22:44 ERROR [http_cmd]: ERROR 13:22:44 INFO [http_cmd]: http status code OK 13:22:44 ERROR --- status code --- 13:22:44 DEBUG --- status code --- 13:22:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:22:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:44 ERROR --- http response --- 13:22:44 DEBUG --- http response --- 13:22:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:22:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:22:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:22:44 DEBUG ----- output ----- 13:22:44 DEBUG { 13:22:44 DEBUG "status": "success", 13:22:44 DEBUG "data": { 13:22:44 DEBUG "resultType": "vector", 13:22:44 DEBUG "result": [ 13:22:44 DEBUG { 13:22:44 DEBUG "metric": { 13:22:44 DEBUG "instance": "10.142.0.107:9100" 13:22:44 DEBUG }, 13:22:44 DEBUG "value": [ 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG [...] 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG "instance": "10.142.0.43:9100" 13:22:44 DEBUG }, 13:22:44 DEBUG "value": [ 13:22:44 DEBUG 1713097364, 13:22:44 DEBUG "7298.5" 13:22:44 DEBUG ] 13:22:44 DEBUG } 13:22:44 DEBUG ] 13:22:44 DEBUG } 13:22:44 DEBUG } 13:22:44 INFO [http_cmd]: http status code OK 13:22:44 DEBUG --- status code --- 13:22:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:44 DEBUG --- http response --- 13:22:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:22:44 DEBUG ----- output ----- 13:22:44 DEBUG { 13:22:44 DEBUG "status": "success", 13:22:44 DEBUG "data": { 13:22:44 DEBUG "resultType": "vector", 13:22:44 DEBUG "result": [ 13:22:44 DEBUG { 13:22:44 DEBUG "metric": { 13:22:44 DEBUG "pod": "ds-idrepo-1" 13:22:44 DEBUG }, 13:22:44 DEBUG "value": [ 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG [...] 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG "pod": "ds-idrepo-0" 13:22:44 DEBUG }, 13:22:44 DEBUG "value": [ 13:22:44 DEBUG 1713097364, 13:22:44 DEBUG "0" 13:22:44 DEBUG ] 13:22:44 DEBUG } 13:22:44 DEBUG ] 13:22:44 DEBUG } 13:22:44 DEBUG } 13:22:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097364" 13:22:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097364" 13:22:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097364" 13:22:44 INFO [http_cmd]: http status code OK 13:22:44 DEBUG --- status code --- 13:22:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:44 DEBUG --- http response --- 13:22:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:44 DEBUG ----- output ----- 13:22:44 DEBUG { 13:22:44 DEBUG "status": "success", 13:22:44 DEBUG "data": { 13:22:44 DEBUG "resultType": "vector", 13:22:44 DEBUG "result": [ 13:22:44 DEBUG { 13:22:44 DEBUG "metric": { 13:22:44 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:22:44 DEBUG }, 13:22:44 DEBUG "value": [ 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG [...] 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG 13:22:44 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:22:44 DEBUG }, 13:22:44 DEBUG "value": [ 13:22:44 DEBUG 1713097364, 13:22:44 DEBUG "0" 13:22:44 DEBUG ] 13:22:44 DEBUG } 13:22:44 DEBUG ] 13:22:44 DEBUG } 13:22:44 DEBUG } 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "ds-idrepo-1" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "ds-cts-1" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097364, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097364, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713097365" 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG { 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG "data": { 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "data": { 13:22:45 DEBUG { 13:22:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG }, 13:22:45 DEBUG "result": [ 13:22:45 DEBUG "data": { 13:22:45 DEBUG "value": [ 13:22:45 DEBUG { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "result": [ 13:22:45 DEBUG 13:22:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:22:45 DEBUG { 13:22:45 DEBUG 13:22:45 DEBUG }, 13:22:45 DEBUG "metric": { 13:22:45 DEBUG [...] 13:22:45 DEBUG "value": [ 13:22:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG }, 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG }, 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "0" 13:22:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:22:45 DEBUG 13:22:45 DEBUG ] 13:22:45 DEBUG }, 13:22:45 DEBUG 13:22:45 DEBUG } 13:22:45 DEBUG "value": [ 13:22:45 DEBUG "pod": "ds-idrepo-2" 13:22:45 DEBUG ] 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG }, 13:22:45 DEBUG } 13:22:45 DEBUG "1.0358909564929129" 13:22:45 DEBUG "value": [ 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG } 13:22:45 DEBUG "5309.367438645006" 13:22:45 DEBUG ] 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "instance": "10.142.0.107:9100" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "instance": "10.142.0.43:9100" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "13026161664" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097365" 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:22:45 DEBUG { 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG { 13:22:45 DEBUG "data": { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "result": [ 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG { 13:22:45 DEBUG "result": [ 13:22:45 DEBUG "metric": { 13:22:45 DEBUG { 13:22:45 DEBUG "pod": "ds-idrepo-1" 13:22:45 DEBUG "metric": { 13:22:45 DEBUG }, 13:22:45 DEBUG "instance": "10.142.0.107:9100" 13:22:45 DEBUG "value": [ 13:22:45 DEBUG }, 13:22:45 DEBUG 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "ds-cts-1" 13:22:45 DEBUG 13:22:45 DEBUG }, 13:22:45 DEBUG "instance": "10.142.0.43:9100" 13:22:45 DEBUG "value": [ 13:22:45 DEBUG }, 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG "0" 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG ] 13:22:45 DEBUG "0.00040000000000001514" 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "ds-idrepo-1" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "ds-idrepo-0" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "instance": "10.142.0.107:9100" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "instance": "10.142.0.43:9100" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "14212.099999999999" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "ds-idrepo-2" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "idm-b85f9d978-rl76f" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "idm-b85f9d978-rl76f" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713097365" 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "5426528256" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "ds-idrepo-1" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "ds-idrepo-0" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0.037" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "ds-idrepo-1" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "ds-idrepo-0" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "pod": "ds-idrepo-1" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "pod": "ds-idrepo-0" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:22:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097365" 13:22:45 INFO [http_cmd]: http status code OK 13:22:45 DEBUG --- status code --- 13:22:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:22:45 DEBUG --- http response --- 13:22:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:22:45 DEBUG ----- output ----- 13:22:45 DEBUG { 13:22:45 DEBUG "status": "success", 13:22:45 DEBUG "data": { 13:22:45 DEBUG "resultType": "vector", 13:22:45 DEBUG "result": [ 13:22:45 DEBUG { 13:22:45 DEBUG "metric": { 13:22:45 DEBUG "instance": "10.142.0.107:9100" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG [...] 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG 13:22:45 DEBUG "instance": "10.142.0.43:9100" 13:22:45 DEBUG }, 13:22:45 DEBUG "value": [ 13:22:45 DEBUG 1713097365, 13:22:45 DEBUG "0.00033333333333333365" 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG ] 13:22:45 DEBUG } 13:22:45 DEBUG } 13:23:02 INFO 13:23:02 INFO [loop_until]: kubectl --namespace=xlou top pods 13:23:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:23:02 INFO [loop_until]: OK (rc = 0) 13:23:02 DEBUG --- stdout --- 13:23:02 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8532Mi am-7d597cf7d6-8vr8f 6m 8603Mi am-7d597cf7d6-9wz9h 7m 8893Mi ds-cts-0 12m 8060Mi ds-cts-1 13m 8376Mi ds-cts-2 13m 8076Mi ds-idrepo-0 11m 4556Mi ds-idrepo-1 10m 4317Mi ds-idrepo-2 10m 4106Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1014Mi idm-b85f9d978-rl76f 4m 1038Mi lodemon-58bdc8d6d4-hh8r9 2m 69Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 982m 5081Mi 13:23:02 DEBUG --- stderr --- 13:23:02 DEBUG 13:23:09 INFO 13:23:09 INFO [loop_until]: kubectl --namespace=xlou top node 13:23:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:23:09 INFO [loop_until]: OK (rc = 0) 13:23:09 DEBUG --- stdout --- 13:23:09 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4873Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 10738Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 10584Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3064Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 63m 0% 6593Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 12067Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2939Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 58m 0% 10050Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5737Mi 5% gke-xlou-cdm-ds-0f601751-j488 67m 0% 10351Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2230Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 84m 0% 10465Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 63m 0% 6228Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1079m 6% 8899Mi 15% 13:23:09 DEBUG --- stderr --- 13:23:09 DEBUG 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713097425" 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097425" 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713097425" 13:23:45 ERROR [http_cmd]: ERROR 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 ERROR --- status code --- 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- status code --- 13:23:45 ERROR --- http response --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:23:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "instance": "10.142.0.107:9100" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "instance": "10.142.0.43:9100" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "7193.9" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "1.0340143254146565" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG { 13:23:45 DEBUG { 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "data": { 13:23:45 DEBUG "data": { 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG "result": [ 13:23:45 DEBUG "result": [ 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG { 13:23:45 DEBUG { 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "ds-idrepo-1" 13:23:45 DEBUG "pod": "ds-idrepo-1" 13:23:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:23:45 DEBUG }, 13:23:45 DEBUG }, 13:23:45 DEBUG }, 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG "value": [ 13:23:45 DEBUG "value": [ 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG [...] 13:23:45 DEBUG [...] 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "ds-idrepo-0" 13:23:45 DEBUG "pod": "ds-cts-1" 13:23:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG }, 13:23:45 DEBUG }, 13:23:45 DEBUG }, 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG "value": [ 13:23:45 DEBUG "value": [ 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG "0" 13:23:45 DEBUG "0" 13:23:45 DEBUG "430.1767897776999" 13:23:45 DEBUG ] 13:23:45 DEBUG ] 13:23:45 DEBUG ] 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG ] 13:23:45 DEBUG ] 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "instance": "10.142.0.107:9100" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "instance": "10.142.0.43:9100" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "13026915328" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097425" 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "ds-idrepo-1" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "ds-cts-1" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "instance": "10.142.0.107:9100" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "instance": "10.142.0.43:9100" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0.00040000000000001514" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "ds-idrepo-1" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "ds-idrepo-0" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "instance": "10.142.0.107:9100" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "instance": "10.142.0.43:9100" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "14141.366666666665" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0.00491199019763918" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "313235380.3605811" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713097425" 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "5421215744" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "ds-idrepo-1" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "ds-idrepo-0" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "am-7d597cf7d6-8ctft" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "am-7d597cf7d6-9wz9h" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0.037" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "ds-idrepo-1" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "ds-idrepo-0" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "pod": "ds-idrepo-1" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "pod": "ds-idrepo-0" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713097425" 13:23:45 INFO [http_cmd]: http status code OK 13:23:45 DEBUG --- status code --- 13:23:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:23:45 DEBUG --- http response --- 13:23:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:23:45 DEBUG ----- output ----- 13:23:45 DEBUG { 13:23:45 DEBUG "status": "success", 13:23:45 DEBUG "data": { 13:23:45 DEBUG "resultType": "vector", 13:23:45 DEBUG "result": [ 13:23:45 DEBUG { 13:23:45 DEBUG "metric": { 13:23:45 DEBUG "instance": "10.142.0.107:9100" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG [...] 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG 13:23:45 DEBUG "instance": "10.142.0.43:9100" 13:23:45 DEBUG }, 13:23:45 DEBUG "value": [ 13:23:45 DEBUG 1713097425, 13:23:45 DEBUG "0.00033333333333333365" 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG ] 13:23:45 DEBUG } 13:23:45 DEBUG } 13:23:56 INFO Finished: True 13:23:56 INFO Waiting for threads to register finish flag 13:24:45 INFO Done. Have a nice day! :) 127.0.0.1 - - [14/Apr/2024 13:24:45] "GET /monitoring/stop HTTP/1.1" 200 - 13:24:49 WARNING File /tmp/lodemon_data-20240414_063908/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6471', ...) 127.0.0.1 - - [14/Apr/2024 13:24:54] "GET /monitoring/process HTTP/1.1" 200 -