==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-7f68c498d9-gbc5j Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Sun, 21 Apr 2024 22:12:06 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=7f68c498d9 skaffold.dev/run-id=c1ae2458-103b-46dd-9597-be35293ba678 Annotations: Status: Running IP: 10.106.46.175 IPs: IP: 10.106.46.175 Controlled By: ReplicaSet/lodemon-7f68c498d9 Containers: lodemon: Container ID: containerd://b60c6e66065c509b3c06929a154de2737f31804163324d3ab085051aedbcce2a Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:03f76e6326d713f4d73dc474e0e25fbbbaecc7c1d8c0590bfd94d68ac0d98c97 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sun, 21 Apr 2024 22:12:07 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m82sp (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-m82sp: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 05:18:04 INFO 05:18:04 INFO [loop_until]: kubectl --namespace=xlou top pods 05:18:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:18:05 INFO [loop_until]: OK (rc = 0) 05:18:05 DEBUG --- stdout --- 05:18:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 23112m 8887Mi am-767f6b67b5-qcqz2 22867m 8940Mi am-767f6b67b5-vbltz 24634m 8881Mi ds-cts-0 20264m 12522Mi ds-cts-1 19929m 12559Mi ds-cts-2 20242m 12531Mi ds-idrepo-0 9494m 3994Mi ds-idrepo-1 9417m 3981Mi ds-idrepo-2 9654m 3965Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 10m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11306m 6219Mi 05:18:05 DEBUG --- stderr --- 05:18:05 DEBUG 05:18:11 INFO 05:18:11 INFO [loop_until]: kubectl --namespace=xlou top node 05:18:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:18:12 INFO [loop_until]: OK (rc = 0) 05:18:12 DEBUG --- stdout --- 05:18:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6227Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10002m 33% 8113Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22867m 76% 11638Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 3351Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19839m 66% 15793Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5204Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 23037m 77% 11870Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19833m 66% 3164Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25272m 84% 11148Mi 9% gke-xlou-cdm-ds-0f601751-j488 10238m 34% 7470Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20137m 67% 17732Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10092m 33% 8045Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20241m 67% 15267Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11713m 73% 16158Mi 27% 05:18:12 DEBUG --- stderr --- 05:18:12 DEBUG 05:18:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713759525" 05:18:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759525" 05:18:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759525" 05:18:45 INFO [http_cmd]: http status code OK 05:18:45 INFO [http_cmd]: http status code OK 05:18:45 ERROR [http_cmd]: ERROR 05:18:45 DEBUG --- status code --- 05:18:45 DEBUG --- status code --- 05:18:45 ERROR --- status code --- 05:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:18:45 DEBUG --- http response --- 05:18:45 DEBUG --- http response --- 05:18:45 ERROR --- http response --- 05:18:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:18:45 DEBUG ----- output ----- 05:18:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:18:45 DEBUG { 05:18:45 DEBUG "status": "success", 05:18:45 DEBUG "data": { 05:18:45 DEBUG "resultType": "vector", 05:18:45 DEBUG "result": [ 05:18:45 DEBUG { 05:18:45 DEBUG "metric": { 05:18:45 DEBUG "pod": "ds-idrepo-0" 05:18:45 DEBUG }, 05:18:45 DEBUG "value": [ 05:18:45 DEBUG 05:18:45 DEBUG 05:18:45 DEBUG 05:18:45 DEBUG [...] 05:18:45 DEBUG 05:18:45 DEBUG 05:18:45 DEBUG 05:18:45 DEBUG "pod": "ds-idrepo-2" 05:18:45 DEBUG }, 05:18:45 DEBUG "value": [ 05:18:45 DEBUG 1713759525, 05:18:45 DEBUG "0" 05:18:45 DEBUG ] 05:18:45 DEBUG } 05:18:45 DEBUG ] 05:18:45 DEBUG } 05:18:45 DEBUG } 05:18:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759525" 05:18:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759525" 05:18:45 INFO [http_cmd]: http status code OK 05:18:45 INFO [http_cmd]: http status code OK 05:18:45 DEBUG --- status code --- 05:18:45 DEBUG --- status code --- 05:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:45 DEBUG --- http response --- 05:18:45 DEBUG --- http response --- 05:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:18:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:18:45 DEBUG ----- output ----- 05:18:45 DEBUG ----- output ----- 05:18:45 DEBUG { 05:18:45 DEBUG { 05:18:45 DEBUG "status": "success", 05:18:45 DEBUG "status": "success", 05:18:45 DEBUG "data": { 05:18:45 DEBUG "data": { 05:18:45 DEBUG "resultType": "vector", 05:18:45 DEBUG "resultType": "vector", 05:18:45 DEBUG "result": [ 05:18:45 DEBUG "result": [ 05:18:45 DEBUG { 05:18:45 DEBUG "metric": { 05:18:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:18:45 DEBUG }, 05:18:45 DEBUG "value": [ 05:18:45 DEBUG 05:18:45 DEBUG { 05:18:45 DEBUG 05:18:45 DEBUG "metric": { 05:18:45 DEBUG 05:18:45 DEBUG "pod": "ds-cts-0" 05:18:45 DEBUG [...] 05:18:45 DEBUG }, 05:18:45 DEBUG 05:18:45 DEBUG "value": [ 05:18:45 DEBUG 05:18:45 DEBUG 05:18:45 DEBUG "pod": "ds-cts-2" 05:18:45 DEBUG }, 05:18:45 DEBUG "value": [ 05:18:45 DEBUG 1713759525, 05:18:45 DEBUG 05:18:45 DEBUG "343016113.32432663" 05:18:45 DEBUG 05:18:45 DEBUG ] 05:18:45 DEBUG 05:18:45 DEBUG [...] 05:18:45 DEBUG 05:18:45 DEBUG 05:18:45 DEBUG 05:18:45 DEBUG "pod": "ds-cts-2" 05:18:45 DEBUG }, 05:18:45 DEBUG "value": [ 05:18:45 DEBUG 1713759525, 05:18:45 DEBUG "0.5754666666666556" 05:18:45 DEBUG ] 05:18:45 DEBUG } 05:18:45 DEBUG ] 05:18:45 DEBUG } 05:18:45 DEBUG } 05:18:45 DEBUG } 05:18:45 DEBUG ] 05:18:45 DEBUG } 05:18:45 DEBUG } 05:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759526" 05:18:46 INFO [http_cmd]: http status code OK 05:18:46 DEBUG --- status code --- 05:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:46 DEBUG --- http response --- 05:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:46 DEBUG ----- output ----- 05:18:46 DEBUG { 05:18:46 DEBUG "status": "success", 05:18:46 DEBUG "data": { 05:18:46 DEBUG "resultType": "vector", 05:18:46 DEBUG "result": [ 05:18:46 DEBUG { 05:18:46 DEBUG "metric": { 05:18:46 DEBUG "instance": "10.142.0.107:9100" 05:18:46 DEBUG }, 05:18:46 DEBUG "value": [ 05:18:46 DEBUG 05:18:46 DEBUG 05:18:46 DEBUG 05:18:46 DEBUG [...] 05:18:46 DEBUG 05:18:46 DEBUG 05:18:46 DEBUG 05:18:46 DEBUG "instance": "10.142.0.43:9100" 05:18:46 DEBUG }, 05:18:46 DEBUG "value": [ 05:18:46 DEBUG 1713759526, 05:18:46 DEBUG "5738992534528" 05:18:46 DEBUG ] 05:18:46 DEBUG } 05:18:46 DEBUG ] 05:18:46 DEBUG } 05:18:46 DEBUG } 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG { 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "data": { 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "result": [ 05:18:47 DEBUG { 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "pod": "ds-cts-0" 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "pod": "ds-cts-2" 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "23.433333333333334" 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG { 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "data": { 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "result": [ 05:18:47 DEBUG { 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "4305.3" 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- status code --- 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:47 DEBUG ----- output ----- 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG { 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG { 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG { 05:18:47 DEBUG "data": { 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG "data": { 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "result": [ 05:18:47 DEBUG "data": { 05:18:47 DEBUG "result": [ 05:18:47 DEBUG { 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG { 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG "metric": { 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG "result": [ 05:18:47 DEBUG "metric": { 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG "pod": "ds-idrepo-0" 05:18:47 DEBUG { 05:18:47 DEBUG { 05:18:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG }, 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "metric": { 05:18:47 DEBUG }, 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG "value": [ 05:18:47 DEBUG "data": { 05:18:47 DEBUG "instance": "10.142.0.107:9100" 05:18:47 DEBUG "value": [ 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:47 DEBUG 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG 05:18:47 DEBUG "result": [ 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG { 05:18:47 DEBUG 05:18:47 DEBUG { 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG [...] 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:18:47 DEBUG [...] 05:18:47 DEBUG "data": { 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "result": [ 05:18:47 DEBUG 05:18:47 DEBUG "pod": "ds-idrepo-2" 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG { 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:18:47 DEBUG 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG "0" 05:18:47 DEBUG "instance": "10.142.0.43:9100" 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG [...] 05:18:47 DEBUG "value": [ 05:18:47 DEBUG ] 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG } 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "1663.6317481807412" 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG ] 05:18:47 DEBUG "0.390233333333155" 05:18:47 DEBUG ] 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:18:47 DEBUG [...] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG } 05:18:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG "value": [ 05:18:47 DEBUG ] 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG } 05:18:47 DEBUG "846.4765" 05:18:47 DEBUG "600702.5" 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713759527" 05:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759527" 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- status code --- 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- status code --- 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG --- status code --- 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- http response --- 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- status code --- 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:47 DEBUG --- status code --- 05:18:47 INFO [http_cmd]: http status code OK 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:47 DEBUG { 05:18:47 DEBUG { 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:18:47 DEBUG { 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG --- status code --- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:47 DEBUG { 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG { 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG { 05:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG { 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG { 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG { 05:18:47 DEBUG "data": { 05:18:47 DEBUG "data": { 05:18:47 DEBUG { 05:18:47 DEBUG "data": { 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG --- http response --- 05:18:47 DEBUG { 05:18:47 DEBUG "data": { 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "data": { 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG { 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "data": { 05:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "result": [ 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "result": [ 05:18:47 DEBUG { 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "data": { 05:18:47 DEBUG "result": [ 05:18:47 DEBUG { 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "result": [ 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG ----- output ----- 05:18:47 DEBUG "data": { 05:18:47 DEBUG { 05:18:47 DEBUG "data": { 05:18:47 DEBUG "data": { 05:18:47 DEBUG { 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "data": { 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:18:47 DEBUG "data": { 05:18:47 DEBUG "result": [ 05:18:47 DEBUG { 05:18:47 DEBUG "result": [ 05:18:47 DEBUG { 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "result": [ 05:18:47 DEBUG }, 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "data": { 05:18:47 DEBUG { 05:18:47 DEBUG "metric": { 05:18:47 DEBUG { 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "result": [ 05:18:47 DEBUG "pod": "ds-idrepo-0" 05:18:47 DEBUG "result": [ 05:18:47 DEBUG "result": [ 05:18:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:18:47 DEBUG "result": [ 05:18:47 DEBUG { 05:18:47 DEBUG "value": [ 05:18:47 DEBUG "result": [ 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:18:47 DEBUG "status": "success", 05:18:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:18:47 DEBUG { 05:18:47 DEBUG }, 05:18:47 DEBUG { 05:18:47 DEBUG { 05:18:47 DEBUG }, 05:18:47 DEBUG { 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "instance": "10.142.0.107:9100" 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG }, 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "value": [ 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "result": [ 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG { 05:18:47 DEBUG "value": [ 05:18:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:18:47 DEBUG 05:18:47 DEBUG "metric": { 05:18:47 DEBUG "pod": "ds-cts-0" 05:18:47 DEBUG 05:18:47 DEBUG "instance": "10.142.0.107:9100" 05:18:47 DEBUG { 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "data": { 05:18:47 DEBUG "metric": { 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG "metric": { 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "resultType": "vector", 05:18:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG [...] 05:18:47 DEBUG }, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG "value": [ 05:18:47 DEBUG "pod": "ds-idrepo-0" 05:18:47 DEBUG }, 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG "result": [ 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG { 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "metric": { 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG "pod": "ds-idrepo-2" 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG "instance": "10.142.0.107:9100" 05:18:47 DEBUG "instance": "10.142.0.43:9100" 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:18:47 DEBUG [...] 05:18:47 DEBUG 05:18:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG }, 05:18:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "value": [ 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG }, 05:18:47 DEBUG [...] 05:18:47 DEBUG "value": [ 05:18:47 DEBUG "value": [ 05:18:47 DEBUG }, 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "3303.5333333333333" 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG ] 05:18:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:18:47 DEBUG }, 05:18:47 DEBUG "pod": "ds-cts-2" 05:18:47 DEBUG "0" 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "3310.4666666666667" 05:18:47 DEBUG "pod": "ds-cts-2" 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG "instance": "10.142.0.43:9100" 05:18:47 DEBUG 05:18:47 DEBUG "101186973.16666666" 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG } 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG }, 05:18:47 DEBUG ] 05:18:47 DEBUG ] 05:18:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG }, 05:18:47 DEBUG 05:18:47 DEBUG 05:18:47 DEBUG ] 05:18:47 DEBUG "3334.133333333333" 05:18:47 DEBUG ] 05:18:47 DEBUG "11.47475810914263" 05:18:47 DEBUG 05:18:47 DEBUG "value": [ 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG }, 05:18:47 DEBUG "0" 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 05:18:47 DEBUG [...] 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG ] 05:18:47 DEBUG ] 05:18:47 DEBUG "value": [ 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "0" 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG "101189315.43333334" 05:18:47 DEBUG 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG }, 05:18:47 DEBUG "0" 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG ] 05:18:47 DEBUG "0" 05:18:47 DEBUG "pod": "ds-cts-2" 05:18:47 DEBUG ] 05:18:47 DEBUG 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG "value": [ 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG }, 05:18:47 DEBUG } 05:18:47 DEBUG 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG "value": [ 05:18:47 DEBUG ] 05:18:47 DEBUG "instance": "10.142.0.43:9100" 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG "6631256064" 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG } 05:18:47 DEBUG }, 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG "0" 05:18:47 DEBUG } 05:18:47 DEBUG "value": [ 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG 1713759527, 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG "0.07499999999998863" 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG ] 05:18:47 DEBUG } 05:18:47 DEBUG } 05:19:05 INFO 05:19:05 INFO [loop_until]: kubectl --namespace=xlou top pods 05:19:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:19:05 INFO [loop_until]: OK (rc = 0) 05:19:05 DEBUG --- stdout --- 05:19:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22449m 8911Mi am-767f6b67b5-qcqz2 23233m 8947Mi am-767f6b67b5-vbltz 25233m 8896Mi ds-cts-0 19817m 12522Mi ds-cts-1 19976m 12557Mi ds-cts-2 19913m 12528Mi ds-idrepo-0 9489m 3994Mi ds-idrepo-1 9491m 3982Mi ds-idrepo-2 9684m 3965Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1243Mi lodemon-7f68c498d9-gbc5j 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11664m 6229Mi 05:19:05 DEBUG --- stderr --- 05:19:05 DEBUG 05:19:12 INFO 05:19:12 INFO [loop_until]: kubectl --namespace=xlou top node 05:19:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:19:12 INFO [loop_until]: OK (rc = 0) 05:19:12 DEBUG --- stdout --- 05:19:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6232Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10098m 33% 8116Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22716m 76% 11659Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3349Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19545m 65% 15797Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 5208Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 23205m 77% 11858Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19876m 66% 3186Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25225m 84% 11188Mi 10% gke-xlou-cdm-ds-0f601751-j488 10234m 34% 7486Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19982m 66% 17737Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10060m 33% 8035Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20249m 67% 15269Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11253m 70% 16160Mi 27% 05:19:12 DEBUG --- stderr --- 05:19:12 DEBUG 05:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713759585" 05:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759585" 05:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759585" 05:19:45 INFO [http_cmd]: http status code OK 05:19:45 DEBUG --- status code --- 05:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:45 DEBUG --- http response --- 05:19:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:45 ERROR [http_cmd]: ERROR 05:19:45 ERROR --- status code --- 05:19:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:19:45 ERROR --- http response --- 05:19:45 INFO [http_cmd]: http status code OK 05:19:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:19:45 DEBUG --- status code --- 05:19:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:45 DEBUG --- http response --- 05:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:45 DEBUG ----- output ----- 05:19:45 DEBUG { 05:19:45 DEBUG "status": "success", 05:19:45 DEBUG "data": { 05:19:45 DEBUG "resultType": "vector", 05:19:45 DEBUG "result": [ 05:19:45 DEBUG { 05:19:45 DEBUG "metric": { 05:19:45 DEBUG "pod": "ds-idrepo-0" 05:19:45 DEBUG }, 05:19:45 DEBUG "value": [ 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG [...] 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG "pod": "ds-idrepo-2" 05:19:45 DEBUG }, 05:19:45 DEBUG "value": [ 05:19:45 DEBUG 1713759585, 05:19:45 DEBUG "0" 05:19:45 DEBUG ] 05:19:45 DEBUG } 05:19:45 DEBUG ] 05:19:45 DEBUG } 05:19:45 DEBUG } 05:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759585" 05:19:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759585" 05:19:45 INFO [http_cmd]: http status code OK 05:19:45 DEBUG --- status code --- 05:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:45 DEBUG --- http response --- 05:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:19:45 DEBUG ----- output ----- 05:19:45 DEBUG { 05:19:45 DEBUG "status": "success", 05:19:45 DEBUG "data": { 05:19:45 DEBUG "resultType": "vector", 05:19:45 DEBUG "result": [ 05:19:45 DEBUG { 05:19:45 DEBUG "metric": { 05:19:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:19:45 DEBUG }, 05:19:45 DEBUG "value": [ 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG [...] 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:19:45 DEBUG }, 05:19:45 DEBUG "value": [ 05:19:45 DEBUG 1713759585, 05:19:45 DEBUG "258193941.47448853" 05:19:45 DEBUG ] 05:19:45 DEBUG } 05:19:45 DEBUG ] 05:19:45 DEBUG } 05:19:45 DEBUG } 05:19:45 INFO [http_cmd]: http status code OK 05:19:45 DEBUG --- status code --- 05:19:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:45 DEBUG --- http response --- 05:19:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:19:45 DEBUG ----- output ----- 05:19:45 DEBUG { 05:19:45 DEBUG "status": "success", 05:19:45 DEBUG "data": { 05:19:45 DEBUG "resultType": "vector", 05:19:45 DEBUG "result": [ 05:19:45 DEBUG { 05:19:45 DEBUG "metric": { 05:19:45 DEBUG "pod": "ds-cts-0" 05:19:45 DEBUG }, 05:19:45 DEBUG "value": [ 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG [...] 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG 05:19:45 DEBUG "pod": "ds-cts-2" 05:19:45 DEBUG }, 05:19:45 DEBUG "value": [ 05:19:45 DEBUG 1713759585, 05:19:45 DEBUG "0.6430999999999965" 05:19:45 DEBUG ] 05:19:45 DEBUG } 05:19:45 DEBUG ] 05:19:45 DEBUG } 05:19:45 DEBUG } 05:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759586" 05:19:46 INFO [http_cmd]: http status code OK 05:19:46 DEBUG --- status code --- 05:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:46 DEBUG --- http response --- 05:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:46 DEBUG ----- output ----- 05:19:46 DEBUG { 05:19:46 DEBUG "status": "success", 05:19:46 DEBUG "data": { 05:19:46 DEBUG "resultType": "vector", 05:19:46 DEBUG "result": [ 05:19:46 DEBUG { 05:19:46 DEBUG "metric": { 05:19:46 DEBUG "instance": "10.142.0.107:9100" 05:19:46 DEBUG }, 05:19:46 DEBUG "value": [ 05:19:46 DEBUG 05:19:46 DEBUG 05:19:46 DEBUG 05:19:46 DEBUG [...] 05:19:46 DEBUG 05:19:46 DEBUG 05:19:46 DEBUG 05:19:46 DEBUG "instance": "10.142.0.43:9100" 05:19:46 DEBUG }, 05:19:46 DEBUG "value": [ 05:19:46 DEBUG 1713759586, 05:19:46 DEBUG "5744185324544" 05:19:46 DEBUG ] 05:19:46 DEBUG } 05:19:46 DEBUG ] 05:19:46 DEBUG } 05:19:46 DEBUG } 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: http status code OK 05:19:47 INFO [http_cmd]: http status code OK 05:19:47 DEBUG --- status code --- 05:19:47 DEBUG --- status code --- 05:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:47 DEBUG --- http response --- 05:19:47 DEBUG --- http response --- 05:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:19:47 DEBUG ----- output ----- 05:19:47 DEBUG ----- output ----- 05:19:47 DEBUG { 05:19:47 DEBUG { 05:19:47 DEBUG "status": "success", 05:19:47 DEBUG "status": "success", 05:19:47 DEBUG "data": { 05:19:47 DEBUG "data": { 05:19:47 DEBUG "resultType": "vector", 05:19:47 DEBUG "resultType": "vector", 05:19:47 DEBUG "result": [ 05:19:47 DEBUG "result": [ 05:19:47 DEBUG { 05:19:47 DEBUG { 05:19:47 DEBUG "metric": { 05:19:47 DEBUG "metric": { 05:19:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:19:47 DEBUG "pod": "ds-cts-0" 05:19:47 DEBUG }, 05:19:47 DEBUG }, 05:19:47 DEBUG "value": [ 05:19:47 DEBUG "value": [ 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG [...] 05:19:47 DEBUG [...] 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:19:47 DEBUG "pod": "ds-cts-2" 05:19:47 DEBUG }, 05:19:47 DEBUG }, 05:19:47 DEBUG "value": [ 05:19:47 DEBUG "value": [ 05:19:47 DEBUG 1713759587, 05:19:47 DEBUG 1713759587, 05:19:47 DEBUG "4335.233333333334" 05:19:47 DEBUG "21.2" 05:19:47 DEBUG ] 05:19:47 DEBUG ] 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 DEBUG ] 05:19:47 DEBUG ] 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: http status code OK 05:19:47 DEBUG --- status code --- 05:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:47 DEBUG --- http response --- 05:19:47 INFO [http_cmd]: http status code OK 05:19:47 DEBUG --- status code --- 05:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:47 DEBUG --- http response --- 05:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:47 DEBUG ----- output ----- 05:19:47 DEBUG ----- output ----- 05:19:47 DEBUG { 05:19:47 INFO [http_cmd]: http status code OK 05:19:47 INFO [http_cmd]: http status code OK 05:19:47 DEBUG { 05:19:47 DEBUG "status": "success", 05:19:47 DEBUG --- status code --- 05:19:47 DEBUG --- status code --- 05:19:47 DEBUG "status": "success", 05:19:47 DEBUG "data": { 05:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:47 INFO [http_cmd]: http status code OK 05:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:47 DEBUG "data": { 05:19:47 DEBUG "resultType": "vector", 05:19:47 DEBUG --- http response --- 05:19:47 DEBUG --- status code --- 05:19:47 DEBUG --- http response --- 05:19:47 DEBUG "resultType": "vector", 05:19:47 DEBUG "result": [ 05:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:47 DEBUG --- http response --- 05:19:47 DEBUG "result": [ 05:19:47 DEBUG { 05:19:47 DEBUG ----- output ----- 05:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:47 DEBUG { 05:19:47 DEBUG "metric": { 05:19:47 DEBUG { 05:19:47 DEBUG ----- output ----- 05:19:47 DEBUG ----- output ----- 05:19:47 DEBUG "metric": { 05:19:47 DEBUG "pod": "ds-idrepo-0" 05:19:47 DEBUG "status": "success", 05:19:47 DEBUG { 05:19:47 DEBUG { 05:19:47 DEBUG "instance": "10.142.0.107:9100" 05:19:47 DEBUG }, 05:19:47 DEBUG "data": { 05:19:47 DEBUG "status": "success", 05:19:47 DEBUG "status": "success", 05:19:47 DEBUG }, 05:19:47 DEBUG "value": [ 05:19:47 DEBUG "resultType": "vector", 05:19:47 DEBUG "data": { 05:19:47 DEBUG "data": { 05:19:47 DEBUG "value": [ 05:19:47 DEBUG 05:19:47 DEBUG "result": [ 05:19:47 DEBUG "resultType": "vector", 05:19:47 DEBUG "resultType": "vector", 05:19:47 DEBUG "result": [ 05:19:47 DEBUG 05:19:47 DEBUG { 05:19:47 DEBUG "result": [ 05:19:47 DEBUG { 05:19:47 DEBUG "metric": { 05:19:47 DEBUG "metric": { 05:19:47 DEBUG 05:19:47 DEBUG { 05:19:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:19:47 DEBUG 05:19:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:19:47 DEBUG 05:19:47 DEBUG "metric": { 05:19:47 DEBUG }, 05:19:47 DEBUG [...] 05:19:47 DEBUG }, 05:19:47 DEBUG 05:19:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:19:47 DEBUG "value": [ 05:19:47 DEBUG 05:19:47 DEBUG "value": [ 05:19:47 DEBUG [...] 05:19:47 DEBUG }, 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG "value": [ 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG "pod": "ds-idrepo-2" 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG [...] 05:19:47 DEBUG }, 05:19:47 DEBUG [...] 05:19:47 DEBUG "instance": "10.142.0.43:9100" 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG "value": [ 05:19:47 DEBUG 05:19:47 DEBUG }, 05:19:47 DEBUG [...] 05:19:47 DEBUG 05:19:47 DEBUG 1713759587, 05:19:47 DEBUG 05:19:47 DEBUG "value": [ 05:19:47 DEBUG 05:19:47 DEBUG 05:19:47 DEBUG "0" 05:19:47 DEBUG 05:19:47 DEBUG 1713759587, 05:19:47 DEBUG 05:19:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:19:47 DEBUG ] 05:19:47 DEBUG } 05:19:47 DEBUG "0.4095666666666868" 05:19:47 DEBUG 05:19:47 DEBUG }, 05:19:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:19:47 DEBUG ] 05:19:47 DEBUG ] 05:19:47 DEBUG "pod": "ds-cts-0" 05:19:47 DEBUG "value": [ 05:19:47 DEBUG }, 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 DEBUG }, 05:19:47 DEBUG 1713759587, 05:19:47 DEBUG "value": [ 05:19:47 DEBUG } 05:19:47 DEBUG ] 05:19:47 DEBUG "value": [ 05:19:47 DEBUG "858.2395" 05:19:47 DEBUG 1713759587, 05:19:47 DEBUG } 05:19:47 DEBUG 1713759587, 05:19:47 DEBUG ] 05:19:47 DEBUG "599883" 05:19:47 DEBUG } 05:19:47 DEBUG "78340703.00061236" 05:19:47 DEBUG } 05:19:47 DEBUG ] 05:19:47 DEBUG ] 05:19:47 DEBUG ] 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 DEBUG ] 05:19:47 DEBUG ] 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 DEBUG } 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713759587" 05:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759587" 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 DEBUG --- status code --- 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 DEBUG --- status code --- 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG --- status code --- 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 INFO [http_cmd]: http status code OK 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG --- status code --- 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG { 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:19:48 DEBUG { 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG --- http response --- 05:19:48 DEBUG { 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG { 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG { 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG "data": { 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG { 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG { 05:19:48 DEBUG { 05:19:48 DEBUG { 05:19:48 DEBUG { 05:19:48 DEBUG { 05:19:48 DEBUG "data": { 05:19:48 DEBUG { 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG ----- output ----- 05:19:48 DEBUG "data": { 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "data": { 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG "data": { 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG { 05:19:48 DEBUG { 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "data": { 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "data": { 05:19:48 DEBUG "data": { 05:19:48 DEBUG "data": { 05:19:48 DEBUG "data": { 05:19:48 DEBUG "data": { 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "data": { 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG "status": "success", 05:19:48 DEBUG "result": [ 05:19:48 DEBUG { 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG { 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "data": { 05:19:48 DEBUG "data": { 05:19:48 DEBUG { 05:19:48 DEBUG "metric": { 05:19:48 DEBUG { 05:19:48 DEBUG "result": [ 05:19:48 DEBUG { 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "metric": { 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "resultType": "vector", 05:19:48 DEBUG "metric": { 05:19:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:19:48 DEBUG }, 05:19:48 DEBUG { 05:19:48 DEBUG "metric": { 05:19:48 DEBUG { 05:19:48 DEBUG "metric": { 05:19:48 DEBUG { 05:19:48 DEBUG { 05:19:48 DEBUG { 05:19:48 DEBUG "instance": "10.142.0.107:9100" 05:19:48 DEBUG { 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "result": [ 05:19:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 05:19:48 DEBUG "metric": { 05:19:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:19:48 DEBUG { 05:19:48 DEBUG "pod": "ds-cts-0" 05:19:48 DEBUG "metric": { 05:19:48 DEBUG "metric": { 05:19:48 DEBUG "metric": { 05:19:48 DEBUG }, 05:19:48 DEBUG "metric": { 05:19:48 DEBUG { 05:19:48 DEBUG { 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG "metric": { 05:19:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:19:48 DEBUG }, 05:19:48 DEBUG "metric": { 05:19:48 DEBUG }, 05:19:48 DEBUG "pod": "ds-idrepo-0" 05:19:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:19:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:19:48 DEBUG "value": [ 05:19:48 DEBUG "pod": "ds-idrepo-0" 05:19:48 DEBUG "metric": { 05:19:48 DEBUG "metric": { 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 05:19:48 DEBUG "instance": "10.142.0.107:9100" 05:19:48 DEBUG }, 05:19:48 DEBUG "value": [ 05:19:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:19:48 DEBUG "value": [ 05:19:48 DEBUG }, 05:19:48 DEBUG }, 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG }, 05:19:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG "value": [ 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG "value": [ 05:19:48 DEBUG "value": [ 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 05:19:48 DEBUG "value": [ 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG "instance": "10.142.0.107:9100" 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG "instance": "10.142.0.43:9100" 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG "pod": "ds-cts-2" 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG "pod": "ds-cts-0" 05:19:48 DEBUG }, 05:19:48 DEBUG "value": [ 05:19:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:19:48 DEBUG 05:19:48 DEBUG [...] 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG "instance": "10.142.0.43:9100" 05:19:48 DEBUG 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 05:19:48 DEBUG }, 05:19:48 DEBUG "value": [ 05:19:48 DEBUG "pod": "ds-cts-0" 05:19:48 DEBUG }, 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG 05:19:48 DEBUG "3338.5333333333333" 05:19:48 DEBUG }, 05:19:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG "pod": "ds-idrepo-2" 05:19:48 DEBUG 05:19:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:19:48 DEBUG }, 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG "101925118.83333334" 05:19:48 DEBUG "value": [ 05:19:48 DEBUG "pod": "ds-cts-2" 05:19:48 DEBUG 05:19:48 DEBUG ] 05:19:48 DEBUG "value": [ 05:19:48 DEBUG }, 05:19:48 DEBUG "3332.866666666667" 05:19:48 DEBUG }, 05:19:48 DEBUG }, 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG ] 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG "3330.9333333333334" 05:19:48 DEBUG }, 05:19:48 DEBUG 05:19:48 DEBUG } 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG "value": [ 05:19:48 DEBUG ] 05:19:48 DEBUG "value": [ 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG "0" 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG "59980564.84996938" 05:19:48 DEBUG } 05:19:48 DEBUG "6622031872" 05:19:48 DEBUG ] 05:19:48 DEBUG "value": [ 05:19:48 DEBUG "instance": "10.142.0.43:9100" 05:19:48 DEBUG ] 05:19:48 DEBUG "101924663.33333334" 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG "0" 05:19:48 DEBUG "0" 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG "20.372004715246145" 05:19:48 DEBUG ] 05:19:48 DEBUG ] 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG }, 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG ] 05:19:48 DEBUG "value": [ 05:19:48 DEBUG ] 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG "0" 05:19:48 DEBUG "value": [ 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG "11.54121305963271" 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG 1713759587, 05:19:48 DEBUG ] 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG "0.08766666666666839" 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG ] 05:19:48 DEBUG } 05:19:48 DEBUG } 05:19:48 DEBUG } 05:20:05 INFO 05:20:05 INFO [loop_until]: kubectl --namespace=xlou top pods 05:20:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:20:05 INFO [loop_until]: OK (rc = 0) 05:20:05 DEBUG --- stdout --- 05:20:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22771m 8887Mi am-767f6b67b5-qcqz2 22906m 8965Mi am-767f6b67b5-vbltz 24752m 8891Mi ds-cts-0 20297m 12527Mi ds-cts-1 20171m 12558Mi ds-cts-2 20604m 12528Mi ds-idrepo-0 9682m 3994Mi ds-idrepo-1 9484m 3982Mi ds-idrepo-2 9617m 3964Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11651m 6225Mi 05:20:05 DEBUG --- stderr --- 05:20:05 DEBUG 05:20:12 INFO 05:20:12 INFO [loop_until]: kubectl --namespace=xlou top node 05:20:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:20:12 INFO [loop_until]: OK (rc = 0) 05:20:12 DEBUG --- stdout --- 05:20:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6231Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9979m 33% 8108Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22734m 76% 11658Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3353Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20433m 68% 15789Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 5198Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22925m 76% 11913Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19869m 66% 3203Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25376m 84% 11165Mi 10% gke-xlou-cdm-ds-0f601751-j488 10234m 34% 7472Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19754m 66% 17734Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10116m 33% 8029Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20351m 68% 15263Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11741m 73% 16161Mi 27% 05:20:12 DEBUG --- stderr --- 05:20:12 DEBUG 05:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759645" 05:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713759645" 05:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759645" 05:20:45 ERROR [http_cmd]: ERROR 05:20:45 INFO [http_cmd]: http status code OK 05:20:45 ERROR --- status code --- 05:20:45 INFO [http_cmd]: http status code OK 05:20:45 DEBUG --- status code --- 05:20:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:20:45 DEBUG --- status code --- 05:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:45 ERROR --- http response --- 05:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:45 DEBUG --- http response --- 05:20:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:20:45 DEBUG --- http response --- 05:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:20:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:45 DEBUG ----- output ----- 05:20:45 DEBUG { 05:20:45 DEBUG "status": "success", 05:20:45 DEBUG "data": { 05:20:45 DEBUG "resultType": "vector", 05:20:45 DEBUG "result": [ 05:20:45 DEBUG { 05:20:45 DEBUG "metric": { 05:20:45 DEBUG "pod": "ds-idrepo-0" 05:20:45 DEBUG }, 05:20:45 DEBUG "value": [ 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG [...] 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG "pod": "ds-idrepo-2" 05:20:45 DEBUG }, 05:20:45 DEBUG "value": [ 05:20:45 DEBUG 1713759645, 05:20:45 DEBUG "0" 05:20:45 DEBUG ] 05:20:45 DEBUG } 05:20:45 DEBUG ] 05:20:45 DEBUG } 05:20:45 DEBUG } 05:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759645" 05:20:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759645" 05:20:45 INFO [http_cmd]: http status code OK 05:20:45 DEBUG --- status code --- 05:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:45 DEBUG --- http response --- 05:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:45 DEBUG ----- output ----- 05:20:45 DEBUG { 05:20:45 DEBUG "status": "success", 05:20:45 DEBUG "data": { 05:20:45 DEBUG "resultType": "vector", 05:20:45 DEBUG "result": [ 05:20:45 DEBUG { 05:20:45 DEBUG "metric": { 05:20:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:20:45 DEBUG }, 05:20:45 DEBUG "value": [ 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG [...] 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:45 DEBUG }, 05:20:45 DEBUG "value": [ 05:20:45 DEBUG 1713759645, 05:20:45 DEBUG "255579822.79308498" 05:20:45 DEBUG ] 05:20:45 DEBUG } 05:20:45 DEBUG ] 05:20:45 DEBUG } 05:20:45 DEBUG } 05:20:45 INFO [http_cmd]: http status code OK 05:20:45 DEBUG --- status code --- 05:20:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:45 DEBUG --- http response --- 05:20:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:20:45 DEBUG ----- output ----- 05:20:45 DEBUG { 05:20:45 DEBUG "status": "success", 05:20:45 DEBUG "data": { 05:20:45 DEBUG "resultType": "vector", 05:20:45 DEBUG "result": [ 05:20:45 DEBUG { 05:20:45 DEBUG "metric": { 05:20:45 DEBUG "pod": "ds-cts-0" 05:20:45 DEBUG }, 05:20:45 DEBUG "value": [ 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG [...] 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG 05:20:45 DEBUG "pod": "ds-cts-2" 05:20:45 DEBUG }, 05:20:45 DEBUG "value": [ 05:20:45 DEBUG 1713759645, 05:20:45 DEBUG "0.5973000000000032" 05:20:45 DEBUG ] 05:20:45 DEBUG } 05:20:45 DEBUG ] 05:20:45 DEBUG } 05:20:45 DEBUG } 05:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759647" 05:20:47 INFO [http_cmd]: http status code OK 05:20:47 DEBUG --- status code --- 05:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:47 DEBUG --- http response --- 05:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:47 DEBUG ----- output ----- 05:20:47 DEBUG { 05:20:47 DEBUG "status": "success", 05:20:47 DEBUG "data": { 05:20:47 DEBUG "resultType": "vector", 05:20:47 DEBUG "result": [ 05:20:47 DEBUG { 05:20:47 DEBUG "metric": { 05:20:47 DEBUG "instance": "10.142.0.107:9100" 05:20:47 DEBUG }, 05:20:47 DEBUG "value": [ 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG [...] 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG "instance": "10.142.0.43:9100" 05:20:47 DEBUG }, 05:20:47 DEBUG "value": [ 05:20:47 DEBUG 1713759647, 05:20:47 DEBUG "5749168999424" 05:20:47 DEBUG ] 05:20:47 DEBUG } 05:20:47 DEBUG ] 05:20:47 DEBUG } 05:20:47 DEBUG } 05:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759647" 05:20:47 INFO [http_cmd]: http status code OK 05:20:47 DEBUG --- status code --- 05:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:47 DEBUG --- http response --- 05:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:47 DEBUG ----- output ----- 05:20:47 DEBUG { 05:20:47 DEBUG "status": "success", 05:20:47 DEBUG "data": { 05:20:47 DEBUG "resultType": "vector", 05:20:47 DEBUG "result": [ 05:20:47 DEBUG { 05:20:47 DEBUG "metric": { 05:20:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:47 DEBUG }, 05:20:47 DEBUG "value": [ 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG [...] 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:20:47 DEBUG }, 05:20:47 DEBUG "value": [ 05:20:47 DEBUG 1713759647, 05:20:47 DEBUG "4316.533333333333" 05:20:47 DEBUG ] 05:20:47 DEBUG } 05:20:47 DEBUG ] 05:20:47 DEBUG } 05:20:47 DEBUG } 05:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759647" 05:20:47 INFO [http_cmd]: http status code OK 05:20:47 DEBUG --- status code --- 05:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:47 DEBUG --- http response --- 05:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:20:47 DEBUG ----- output ----- 05:20:47 DEBUG { 05:20:47 DEBUG "status": "success", 05:20:47 DEBUG "data": { 05:20:47 DEBUG "resultType": "vector", 05:20:47 DEBUG "result": [ 05:20:47 DEBUG { 05:20:47 DEBUG "metric": { 05:20:47 DEBUG "pod": "ds-cts-0" 05:20:47 DEBUG }, 05:20:47 DEBUG "value": [ 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG [...] 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG "pod": "ds-cts-2" 05:20:47 DEBUG }, 05:20:47 DEBUG "value": [ 05:20:47 DEBUG 1713759647, 05:20:47 DEBUG "22" 05:20:47 DEBUG ] 05:20:47 DEBUG } 05:20:47 DEBUG ] 05:20:47 DEBUG } 05:20:47 DEBUG } 05:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759647" 05:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759647" 05:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713759647" 05:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759647" 05:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759647" 05:20:47 INFO [http_cmd]: http status code OK 05:20:47 INFO [http_cmd]: http status code OK 05:20:47 INFO [http_cmd]: http status code OK 05:20:47 DEBUG --- status code --- 05:20:47 DEBUG --- status code --- 05:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:47 DEBUG --- http response --- 05:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:47 DEBUG --- http response --- 05:20:47 DEBUG ----- output ----- 05:20:47 DEBUG --- status code --- 05:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:47 DEBUG { 05:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:47 DEBUG ----- output ----- 05:20:47 DEBUG "status": "success", 05:20:47 DEBUG --- http response --- 05:20:47 DEBUG { 05:20:47 DEBUG "data": { 05:20:47 DEBUG "status": "success", 05:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:47 DEBUG "resultType": "vector", 05:20:47 DEBUG "data": { 05:20:47 DEBUG ----- output ----- 05:20:47 DEBUG "result": [ 05:20:47 DEBUG "resultType": "vector", 05:20:47 DEBUG { 05:20:47 DEBUG { 05:20:47 DEBUG "result": [ 05:20:47 DEBUG "status": "success", 05:20:47 DEBUG "data": { 05:20:47 DEBUG { 05:20:47 DEBUG "resultType": "vector", 05:20:47 DEBUG "metric": { 05:20:47 DEBUG "metric": { 05:20:47 DEBUG "result": [ 05:20:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:47 DEBUG }, 05:20:47 DEBUG { 05:20:47 DEBUG "value": [ 05:20:47 DEBUG 05:20:47 DEBUG "metric": { 05:20:47 DEBUG "pod": "ds-idrepo-0" 05:20:47 DEBUG 05:20:47 DEBUG "instance": "10.142.0.107:9100" 05:20:47 DEBUG }, 05:20:47 DEBUG 05:20:47 DEBUG }, 05:20:47 DEBUG "value": [ 05:20:47 DEBUG [...] 05:20:47 DEBUG "value": [ 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 INFO [http_cmd]: http status code OK 05:20:47 DEBUG [...] 05:20:47 INFO [http_cmd]: http status code OK 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG --- status code --- 05:20:47 DEBUG 05:20:47 DEBUG --- status code --- 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:47 DEBUG 05:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:47 DEBUG 05:20:47 DEBUG [...] 05:20:47 DEBUG --- http response --- 05:20:47 DEBUG 05:20:47 DEBUG --- http response --- 05:20:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:20:47 DEBUG 05:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:20:47 DEBUG "pod": "ds-idrepo-2" 05:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:47 DEBUG }, 05:20:47 DEBUG 05:20:47 DEBUG ----- output ----- 05:20:47 DEBUG }, 05:20:47 DEBUG ----- output ----- 05:20:47 DEBUG "value": [ 05:20:47 DEBUG 05:20:47 DEBUG { 05:20:47 DEBUG "value": [ 05:20:47 DEBUG { 05:20:47 DEBUG 1713759647, 05:20:47 DEBUG "instance": "10.142.0.43:9100" 05:20:47 DEBUG "status": "success", 05:20:47 DEBUG 1713759647, 05:20:47 DEBUG "status": "success", 05:20:47 DEBUG "868.9665" 05:20:47 DEBUG }, 05:20:47 DEBUG "data": { 05:20:47 DEBUG "0" 05:20:47 DEBUG "data": { 05:20:47 DEBUG ] 05:20:47 DEBUG "value": [ 05:20:47 DEBUG "resultType": "vector", 05:20:47 DEBUG ] 05:20:47 DEBUG "resultType": "vector", 05:20:47 DEBUG } 05:20:47 DEBUG 1713759647, 05:20:47 DEBUG "result": [ 05:20:47 DEBUG } 05:20:47 DEBUG "result": [ 05:20:47 DEBUG ] 05:20:47 DEBUG "0.3560333333331471" 05:20:47 DEBUG { 05:20:47 DEBUG ] 05:20:47 DEBUG } 05:20:47 DEBUG } 05:20:47 DEBUG ] 05:20:47 DEBUG "metric": { 05:20:47 DEBUG } 05:20:47 DEBUG } 05:20:47 DEBUG } 05:20:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:20:47 DEBUG { 05:20:47 DEBUG ] 05:20:47 DEBUG }, 05:20:47 DEBUG "metric": { 05:20:47 DEBUG } 05:20:47 DEBUG "value": [ 05:20:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:47 DEBUG }, 05:20:47 DEBUG 05:20:47 DEBUG } 05:20:47 DEBUG "value": [ 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG [...] 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG [...] 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG 05:20:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:47 DEBUG 05:20:47 DEBUG }, 05:20:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:20:47 DEBUG "value": [ 05:20:47 DEBUG }, 05:20:47 DEBUG 1713759647, 05:20:47 DEBUG "value": [ 05:20:47 DEBUG "424.4102264426589" 05:20:47 DEBUG 1713759647, 05:20:47 DEBUG ] 05:20:47 DEBUG "600425" 05:20:47 DEBUG } 05:20:47 DEBUG ] 05:20:47 DEBUG ] 05:20:47 DEBUG } 05:20:47 DEBUG } 05:20:47 DEBUG ] 05:20:47 DEBUG } 05:20:47 DEBUG } 05:20:47 DEBUG } 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759648" 05:20:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759648" 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG --- status code --- 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG --- status code --- 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:48 DEBUG ----- output ----- 05:20:48 INFO [http_cmd]: http status code OK 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG { 05:20:48 DEBUG --- status code --- 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:20:48 DEBUG { 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:48 DEBUG { 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:20:48 DEBUG { 05:20:48 DEBUG { 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:48 DEBUG { 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG { 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG "data": { 05:20:48 DEBUG --- http response --- 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG "data": { 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG { 05:20:48 DEBUG "data": { 05:20:48 DEBUG { 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG { 05:20:48 DEBUG "data": { 05:20:48 DEBUG { 05:20:48 DEBUG "data": { 05:20:48 DEBUG "data": { 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:48 DEBUG "data": { 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG "result": [ 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG { 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG "result": [ 05:20:48 DEBUG ----- output ----- 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG { 05:20:48 DEBUG { 05:20:48 DEBUG "data": { 05:20:48 DEBUG "result": [ 05:20:48 DEBUG { 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG "data": { 05:20:48 DEBUG "result": [ 05:20:48 DEBUG "data": { 05:20:48 DEBUG "result": [ 05:20:48 DEBUG "result": [ 05:20:48 DEBUG { 05:20:48 DEBUG { 05:20:48 DEBUG "result": [ 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "data": { 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG { 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "data": { 05:20:48 DEBUG { 05:20:48 DEBUG { 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "status": "success", 05:20:48 DEBUG { 05:20:48 DEBUG "instance": "10.142.0.107:9100" 05:20:48 DEBUG "data": { 05:20:48 DEBUG "result": [ 05:20:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG "result": [ 05:20:48 DEBUG "pod": "ds-idrepo-0" 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "pod": "ds-cts-0" 05:20:48 DEBUG "data": { 05:20:48 DEBUG "metric": { 05:20:48 DEBUG }, 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG { 05:20:48 DEBUG }, 05:20:48 DEBUG "result": [ 05:20:48 DEBUG { 05:20:48 DEBUG }, 05:20:48 DEBUG "result": [ 05:20:48 DEBUG "result": [ 05:20:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:48 DEBUG }, 05:20:48 DEBUG "resultType": "vector", 05:20:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:20:48 DEBUG "value": [ 05:20:48 DEBUG "result": [ 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "value": [ 05:20:48 DEBUG { 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "value": [ 05:20:48 DEBUG { 05:20:48 DEBUG { 05:20:48 DEBUG }, 05:20:48 DEBUG }, 05:20:48 DEBUG "value": [ 05:20:48 DEBUG "result": [ 05:20:48 DEBUG }, 05:20:48 DEBUG 05:20:48 DEBUG { 05:20:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:48 DEBUG 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "instance": "10.142.0.107:9100" 05:20:48 DEBUG 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "metric": { 05:20:48 DEBUG "value": [ 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG { 05:20:48 DEBUG "metric": { 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG }, 05:20:48 DEBUG 05:20:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:48 DEBUG }, 05:20:48 DEBUG 05:20:48 DEBUG "pod": "ds-idrepo-0" 05:20:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG "metric": { 05:20:48 DEBUG [...] 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG "value": [ 05:20:48 DEBUG }, 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG }, 05:20:48 DEBUG }, 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG "instance": "10.142.0.107:9100" 05:20:48 DEBUG [...] 05:20:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG [...] 05:20:48 DEBUG 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG [...] 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG }, 05:20:48 DEBUG 05:20:48 DEBUG }, 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG [...] 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG [...] 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG "instance": "10.142.0.43:9100" 05:20:48 DEBUG [...] 05:20:48 DEBUG 05:20:48 DEBUG [...] 05:20:48 DEBUG 05:20:48 DEBUG [...] 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG "pod": "ds-cts-2" 05:20:48 DEBUG }, 05:20:48 DEBUG }, 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG [...] 05:20:48 DEBUG 05:20:48 DEBUG "pod": "ds-idrepo-2" 05:20:48 DEBUG [...] 05:20:48 DEBUG 05:20:48 DEBUG [...] 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG [...] 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG 05:20:48 DEBUG }, 05:20:48 DEBUG 05:20:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:20:48 DEBUG 05:20:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:20:48 DEBUG "102368031.53333333" 05:20:48 DEBUG 05:20:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:20:48 DEBUG 05:20:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:48 DEBUG 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG }, 05:20:48 DEBUG 05:20:48 DEBUG }, 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG 05:20:48 DEBUG }, 05:20:48 DEBUG ] 05:20:48 DEBUG [...] 05:20:48 DEBUG }, 05:20:48 DEBUG }, 05:20:48 DEBUG "value": [ 05:20:48 DEBUG "instance": "10.142.0.43:9100" 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 05:20:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:20:48 DEBUG "0" 05:20:48 DEBUG 05:20:48 DEBUG "value": [ 05:20:48 DEBUG } 05:20:48 DEBUG 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG 05:20:48 DEBUG }, 05:20:48 DEBUG "0" 05:20:48 DEBUG "pod": "ds-cts-2" 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG }, 05:20:48 DEBUG ] 05:20:48 DEBUG "value": [ 05:20:48 DEBUG "instance": "10.142.0.43:9100" 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG ] 05:20:48 DEBUG 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG "0" 05:20:48 DEBUG 05:20:48 DEBUG "value": [ 05:20:48 DEBUG ] 05:20:48 DEBUG }, 05:20:48 DEBUG "3353.9333333333334" 05:20:48 DEBUG "value": [ 05:20:48 DEBUG } 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG }, 05:20:48 DEBUG "6623170560" 05:20:48 DEBUG } 05:20:48 DEBUG 05:20:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG } 05:20:48 DEBUG "value": [ 05:20:48 DEBUG ] 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG ] 05:20:48 DEBUG "3332.9333333333334" 05:20:48 DEBUG "value": [ 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG }, 05:20:48 DEBUG "value": [ 05:20:48 DEBUG "3355.6666666666665" 05:20:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:20:48 DEBUG }, 05:20:48 DEBUG ] 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG } 05:20:48 DEBUG "11.518854806118899" 05:20:48 DEBUG } 05:20:48 DEBUG ] 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG } 05:20:48 DEBUG ] 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG ] 05:20:48 DEBUG "value": [ 05:20:48 DEBUG 1713759648, 05:20:48 DEBUG "102365301.4" 05:20:48 DEBUG "0" 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG "0.066999999999994" 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG "12.616693185473261" 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG "0" 05:20:48 DEBUG ] 05:20:48 DEBUG ] 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG ] 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG ] 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG ] 05:20:48 DEBUG ] 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG ] 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG ] 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:20:48 DEBUG } 05:21:05 INFO 05:21:05 INFO [loop_until]: kubectl --namespace=xlou top pods 05:21:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:21:05 INFO [loop_until]: OK (rc = 0) 05:21:05 DEBUG --- stdout --- 05:21:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22459m 8886Mi am-767f6b67b5-qcqz2 22963m 8938Mi am-767f6b67b5-vbltz 24492m 8882Mi ds-cts-0 20302m 12522Mi ds-cts-1 20307m 12559Mi ds-cts-2 20151m 12529Mi ds-idrepo-0 9236m 3994Mi ds-idrepo-1 9424m 3982Mi ds-idrepo-2 9793m 3964Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11503m 6219Mi 05:21:05 DEBUG --- stderr --- 05:21:05 DEBUG 05:21:12 INFO 05:21:12 INFO [loop_until]: kubectl --namespace=xlou top node 05:21:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:21:12 INFO [loop_until]: OK (rc = 0) 05:21:12 DEBUG --- stdout --- 05:21:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6227Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9978m 33% 8121Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22779m 76% 11643Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3339Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 20195m 67% 15759Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5225Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 23009m 77% 11858Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19779m 66% 3173Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25230m 84% 11144Mi 9% gke-xlou-cdm-ds-0f601751-j488 10243m 34% 7471Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19190m 64% 17746Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10065m 33% 8032Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20448m 68% 15261Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11722m 73% 16161Mi 27% 05:21:12 DEBUG --- stderr --- 05:21:12 DEBUG 05:21:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713759705" 05:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759706" 05:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759706" 05:21:46 ERROR [http_cmd]: ERROR 05:21:46 ERROR --- status code --- 05:21:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:21:46 ERROR --- http response --- 05:21:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:21:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:21:46 INFO [http_cmd]: http status code OK 05:21:46 DEBUG --- status code --- 05:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:46 DEBUG --- http response --- 05:21:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:46 INFO [http_cmd]: http status code OK 05:21:46 DEBUG --- status code --- 05:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:46 DEBUG --- http response --- 05:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:46 DEBUG ----- output ----- 05:21:46 DEBUG { 05:21:46 DEBUG "status": "success", 05:21:46 DEBUG "data": { 05:21:46 DEBUG "resultType": "vector", 05:21:46 DEBUG "result": [ 05:21:46 DEBUG { 05:21:46 DEBUG "metric": { 05:21:46 DEBUG "pod": "ds-idrepo-0" 05:21:46 DEBUG }, 05:21:46 DEBUG "value": [ 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG [...] 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG "pod": "ds-idrepo-2" 05:21:46 DEBUG }, 05:21:46 DEBUG "value": [ 05:21:46 DEBUG 1713759706, 05:21:46 DEBUG "0" 05:21:46 DEBUG ] 05:21:46 DEBUG } 05:21:46 DEBUG ] 05:21:46 DEBUG } 05:21:46 DEBUG } 05:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759706" 05:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759706" 05:21:46 INFO [http_cmd]: http status code OK 05:21:46 DEBUG --- status code --- 05:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:46 DEBUG --- http response --- 05:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:21:46 DEBUG ----- output ----- 05:21:46 DEBUG { 05:21:46 DEBUG "status": "success", 05:21:46 DEBUG "data": { 05:21:46 DEBUG "resultType": "vector", 05:21:46 DEBUG "result": [ 05:21:46 DEBUG { 05:21:46 DEBUG "metric": { 05:21:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:21:46 DEBUG }, 05:21:46 DEBUG "value": [ 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG [...] 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG "pod": "ds-cts-2" 05:21:46 DEBUG }, 05:21:46 DEBUG "value": [ 05:21:46 DEBUG 1713759706, 05:21:46 DEBUG "0" 05:21:46 DEBUG ] 05:21:46 DEBUG } 05:21:46 DEBUG ] 05:21:46 DEBUG } 05:21:46 DEBUG } 05:21:46 INFO [http_cmd]: http status code OK 05:21:46 DEBUG --- status code --- 05:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:46 DEBUG --- http response --- 05:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:21:46 DEBUG ----- output ----- 05:21:46 DEBUG { 05:21:46 DEBUG "status": "success", 05:21:46 DEBUG "data": { 05:21:46 DEBUG "resultType": "vector", 05:21:46 DEBUG "result": [ 05:21:46 DEBUG { 05:21:46 DEBUG "metric": { 05:21:46 DEBUG "pod": "ds-cts-0" 05:21:46 DEBUG }, 05:21:46 DEBUG "value": [ 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG [...] 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG 05:21:46 DEBUG "pod": "ds-cts-2" 05:21:46 DEBUG }, 05:21:46 DEBUG "value": [ 05:21:46 DEBUG 1713759706, 05:21:46 DEBUG "0.6195999999999988" 05:21:46 DEBUG ] 05:21:46 DEBUG } 05:21:46 DEBUG ] 05:21:46 DEBUG } 05:21:46 DEBUG } 05:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759707" 05:21:47 INFO [http_cmd]: http status code OK 05:21:47 DEBUG --- status code --- 05:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:47 DEBUG --- http response --- 05:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:47 DEBUG ----- output ----- 05:21:47 DEBUG { 05:21:47 DEBUG "status": "success", 05:21:47 DEBUG "data": { 05:21:47 DEBUG "resultType": "vector", 05:21:47 DEBUG "result": [ 05:21:47 DEBUG { 05:21:47 DEBUG "metric": { 05:21:47 DEBUG "instance": "10.142.0.107:9100" 05:21:47 DEBUG }, 05:21:47 DEBUG "value": [ 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG [...] 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG "instance": "10.142.0.43:9100" 05:21:47 DEBUG }, 05:21:47 DEBUG "value": [ 05:21:47 DEBUG 1713759707, 05:21:47 DEBUG "5754145545216" 05:21:47 DEBUG ] 05:21:47 DEBUG } 05:21:47 DEBUG ] 05:21:47 DEBUG } 05:21:47 DEBUG } 05:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759707" 05:21:47 INFO [http_cmd]: http status code OK 05:21:47 DEBUG --- status code --- 05:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:47 DEBUG --- http response --- 05:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:47 DEBUG ----- output ----- 05:21:47 DEBUG { 05:21:47 DEBUG "status": "success", 05:21:47 DEBUG "data": { 05:21:47 DEBUG "resultType": "vector", 05:21:47 DEBUG "result": [ 05:21:47 DEBUG { 05:21:47 DEBUG "metric": { 05:21:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:21:47 DEBUG }, 05:21:47 DEBUG "value": [ 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG [...] 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:21:47 DEBUG }, 05:21:47 DEBUG "value": [ 05:21:47 DEBUG 1713759707, 05:21:47 DEBUG "4317.933333333333" 05:21:47 DEBUG ] 05:21:47 DEBUG } 05:21:47 DEBUG ] 05:21:47 DEBUG } 05:21:47 DEBUG } 05:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759707" 05:21:47 INFO [http_cmd]: http status code OK 05:21:47 DEBUG --- status code --- 05:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:47 DEBUG --- http response --- 05:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:21:47 DEBUG ----- output ----- 05:21:47 DEBUG { 05:21:47 DEBUG "status": "success", 05:21:47 DEBUG "data": { 05:21:47 DEBUG "resultType": "vector", 05:21:47 DEBUG "result": [ 05:21:47 DEBUG { 05:21:47 DEBUG "metric": { 05:21:47 DEBUG "pod": "ds-cts-0" 05:21:47 DEBUG }, 05:21:47 DEBUG "value": [ 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG [...] 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG 05:21:47 DEBUG "pod": "ds-cts-2" 05:21:47 DEBUG }, 05:21:47 DEBUG "value": [ 05:21:47 DEBUG 1713759707, 05:21:47 DEBUG "21.333333333333332" 05:21:47 DEBUG ] 05:21:47 DEBUG } 05:21:47 DEBUG ] 05:21:47 DEBUG } 05:21:47 DEBUG } 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:48 DEBUG { 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG "status": "success", 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG "data": { 05:21:48 DEBUG { 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "data": { 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG { 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "metric": { 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG { 05:21:48 DEBUG "pod": "ds-idrepo-0" 05:21:48 DEBUG { 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG "metric": { 05:21:48 DEBUG }, 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "data": { 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG }, 05:21:48 DEBUG 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG "result": [ 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG { 05:21:48 DEBUG { 05:21:48 DEBUG { 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG "metric": { 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG "data": { 05:21:48 DEBUG "instance": "10.142.0.107:9100" 05:21:48 DEBUG "data": { 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG }, 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "pod": "ds-idrepo-2" 05:21:48 DEBUG 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "result": [ 05:21:48 DEBUG }, 05:21:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:21:48 DEBUG { 05:21:48 DEBUG 05:21:48 DEBUG { 05:21:48 DEBUG "value": [ 05:21:48 DEBUG }, 05:21:48 DEBUG "metric": { 05:21:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:21:48 DEBUG "metric": { 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:21:48 DEBUG "0" 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG 05:21:48 DEBUG "value": [ 05:21:48 DEBUG }, 05:21:48 DEBUG ] 05:21:48 DEBUG "878.425" 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG "instance": "10.142.0.43:9100" 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG } 05:21:48 DEBUG }, 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:21:48 DEBUG 05:21:48 DEBUG "0.3623666666666243" 05:21:48 DEBUG }, 05:21:48 DEBUG "value": [ 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG ] 05:21:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:21:48 DEBUG }, 05:21:48 DEBUG "value": [ 05:21:48 DEBUG } 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG "600657.5" 05:21:48 DEBUG } 05:21:48 DEBUG "1758.317235458253" 05:21:48 DEBUG ] 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713759708" 05:21:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759708" 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG --- status code --- 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG --- status code --- 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG --- status code --- 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG --- status code --- 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:48 DEBUG --- http response --- 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:48 DEBUG --- http response --- 05:21:48 INFO [http_cmd]: http status code OK 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:48 DEBUG --- status code --- 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG { 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG { 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG { 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG { 05:21:48 DEBUG { 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG { 05:21:48 DEBUG { 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG { 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG { 05:21:48 DEBUG --- http response --- 05:21:48 DEBUG "data": { 05:21:48 DEBUG { 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG "data": { 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG { 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:48 DEBUG "data": { 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG "data": { 05:21:48 DEBUG "data": { 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG "data": { 05:21:48 DEBUG "data": { 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "data": { 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG { 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "data": { 05:21:48 DEBUG ----- output ----- 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "data": { 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "data": { 05:21:48 DEBUG { 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG { 05:21:48 DEBUG { 05:21:48 DEBUG "metric": { 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "result": [ 05:21:48 DEBUG { 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG { 05:21:48 DEBUG "data": { 05:21:48 DEBUG { 05:21:48 DEBUG { 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "status": "success", 05:21:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG { 05:21:48 DEBUG { 05:21:48 DEBUG "metric": { 05:21:48 DEBUG { 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "data": { 05:21:48 DEBUG "metric": { 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "metric": { 05:21:48 DEBUG "metric": { 05:21:48 DEBUG { 05:21:48 DEBUG "data": { 05:21:48 DEBUG }, 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "metric": { 05:21:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:21:48 DEBUG "metric": { 05:21:48 DEBUG "metric": { 05:21:48 DEBUG { 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "pod": "ds-cts-0" 05:21:48 DEBUG "result": [ 05:21:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:21:48 DEBUG "pod": "ds-idrepo-0" 05:21:48 DEBUG "metric": { 05:21:48 DEBUG "resultType": "vector", 05:21:48 DEBUG "value": [ 05:21:48 DEBUG { 05:21:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:21:48 DEBUG }, 05:21:48 DEBUG "pod": "ds-idrepo-0" 05:21:48 DEBUG "instance": "10.142.0.107:9100" 05:21:48 DEBUG }, 05:21:48 DEBUG "result": [ 05:21:48 DEBUG }, 05:21:48 DEBUG { 05:21:48 DEBUG }, 05:21:48 DEBUG }, 05:21:48 DEBUG "instance": "10.142.0.107:9100" 05:21:48 DEBUG "result": [ 05:21:48 DEBUG 05:21:48 DEBUG "metric": { 05:21:48 DEBUG }, 05:21:48 DEBUG "value": [ 05:21:48 DEBUG }, 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "metric": { 05:21:48 DEBUG { 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "metric": { 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG { 05:21:48 DEBUG 05:21:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:21:48 DEBUG "metric": { 05:21:48 DEBUG 05:21:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "metric": { 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG "instance": "10.142.0.107:9100" 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:21:48 DEBUG [...] 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "value": [ 05:21:48 DEBUG }, 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:21:48 DEBUG "pod": "ds-cts-2" 05:21:48 DEBUG }, 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "instance": "10.142.0.43:9100" 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:21:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG 05:21:48 DEBUG 05:21:48 DEBUG "instance": "10.142.0.43:9100" 05:21:48 DEBUG }, 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG [...] 05:21:48 DEBUG 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG "pod": "ds-cts-2" 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG }, 05:21:48 DEBUG }, 05:21:48 DEBUG 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 05:21:48 DEBUG "pod": "ds-cts-2" 05:21:48 DEBUG }, 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "6624120832" 05:21:48 DEBUG 05:21:48 DEBUG "instance": "10.142.0.43:9100" 05:21:48 DEBUG "0" 05:21:48 DEBUG }, 05:21:48 DEBUG "0" 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "pod": "ds-idrepo-2" 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG ] 05:21:48 DEBUG 05:21:48 DEBUG }, 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG }, 05:21:48 DEBUG "3340.6" 05:21:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:21:48 DEBUG "value": [ 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG "0.06799999999999878" 05:21:48 DEBUG ] 05:21:48 DEBUG 05:21:48 DEBUG "value": [ 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG "value": [ 05:21:48 DEBUG "3322.1666666666665" 05:21:48 DEBUG "0" 05:21:48 DEBUG "value": [ 05:21:48 DEBUG ] 05:21:48 DEBUG }, 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG "101465291.3" 05:21:48 DEBUG ] 05:21:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG ] 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG "0" 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG }, 05:21:48 DEBUG "101466791.86666667" 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG "value": [ 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG ] 05:21:48 DEBUG ] 05:21:48 DEBUG "value": [ 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG "0" 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG "0" 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG 1713759708, 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG "11.558334782196141" 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG "3320.133333333333" 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG ] 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:21:48 DEBUG } 05:22:05 INFO 05:22:05 INFO [loop_until]: kubectl --namespace=xlou top pods 05:22:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:22:05 INFO [loop_until]: OK (rc = 0) 05:22:05 DEBUG --- stdout --- 05:22:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22692m 8884Mi am-767f6b67b5-qcqz2 22862m 8980Mi am-767f6b67b5-vbltz 24054m 8901Mi ds-cts-0 19904m 12520Mi ds-cts-1 19492m 12555Mi ds-cts-2 20299m 12530Mi ds-idrepo-0 9432m 3994Mi ds-idrepo-1 9510m 3982Mi ds-idrepo-2 9428m 3965Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 10m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11482m 6223Mi 05:22:05 DEBUG --- stderr --- 05:22:05 DEBUG 05:22:12 INFO 05:22:12 INFO [loop_until]: kubectl --namespace=xlou top node 05:22:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:22:12 INFO [loop_until]: OK (rc = 0) 05:22:12 DEBUG --- stdout --- 05:22:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6224Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10034m 33% 8119Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22796m 76% 11643Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3336Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 20182m 67% 15781Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5205Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 23270m 77% 11889Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19835m 66% 3168Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25448m 85% 11160Mi 9% gke-xlou-cdm-ds-0f601751-j488 10263m 34% 7481Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20056m 67% 17737Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10096m 33% 8042Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20236m 67% 15267Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11801m 74% 16162Mi 27% 05:22:12 DEBUG --- stderr --- 05:22:12 DEBUG 05:22:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713759766" 05:22:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759766" 05:22:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759766" 05:22:46 INFO [http_cmd]: http status code OK 05:22:46 ERROR [http_cmd]: ERROR 05:22:46 INFO [http_cmd]: http status code OK 05:22:46 DEBUG --- status code --- 05:22:46 ERROR --- status code --- 05:22:46 DEBUG --- status code --- 05:22:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:22:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:46 DEBUG --- http response --- 05:22:46 ERROR --- http response --- 05:22:46 DEBUG --- http response --- 05:22:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:22:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:22:46 DEBUG ----- output ----- 05:22:46 DEBUG { 05:22:46 DEBUG "status": "success", 05:22:46 DEBUG "data": { 05:22:46 DEBUG "resultType": "vector", 05:22:46 DEBUG "result": [ 05:22:46 DEBUG { 05:22:46 DEBUG "metric": { 05:22:46 DEBUG "pod": "ds-idrepo-0" 05:22:46 DEBUG }, 05:22:46 DEBUG "value": [ 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG [...] 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG "pod": "ds-idrepo-2" 05:22:46 DEBUG }, 05:22:46 DEBUG "value": [ 05:22:46 DEBUG 1713759766, 05:22:46 DEBUG "0" 05:22:46 DEBUG ] 05:22:46 DEBUG } 05:22:46 DEBUG ] 05:22:46 DEBUG } 05:22:46 DEBUG } 05:22:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759766" 05:22:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759766" 05:22:46 INFO [http_cmd]: http status code OK 05:22:46 INFO [http_cmd]: http status code OK 05:22:46 DEBUG --- status code --- 05:22:46 DEBUG --- status code --- 05:22:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:46 DEBUG --- http response --- 05:22:46 DEBUG --- http response --- 05:22:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:22:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:22:46 DEBUG ----- output ----- 05:22:46 DEBUG ----- output ----- 05:22:46 DEBUG { 05:22:46 DEBUG { 05:22:46 DEBUG "status": "success", 05:22:46 DEBUG "status": "success", 05:22:46 DEBUG "data": { 05:22:46 DEBUG "resultType": "vector", 05:22:46 DEBUG "result": [ 05:22:46 DEBUG { 05:22:46 DEBUG "metric": { 05:22:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:22:46 DEBUG }, 05:22:46 DEBUG "value": [ 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG [...] 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG "pod": "ds-cts-2" 05:22:46 DEBUG }, 05:22:46 DEBUG "value": [ 05:22:46 DEBUG 1713759766, 05:22:46 DEBUG "16871.327549755777" 05:22:46 DEBUG ] 05:22:46 DEBUG } 05:22:46 DEBUG ] 05:22:46 DEBUG } 05:22:46 DEBUG } 05:22:46 DEBUG "data": { 05:22:46 DEBUG "resultType": "vector", 05:22:46 DEBUG "result": [ 05:22:46 DEBUG { 05:22:46 DEBUG "metric": { 05:22:46 DEBUG "pod": "ds-cts-0" 05:22:46 DEBUG }, 05:22:46 DEBUG "value": [ 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG [...] 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG 05:22:46 DEBUG "pod": "ds-cts-2" 05:22:46 DEBUG }, 05:22:46 DEBUG "value": [ 05:22:46 DEBUG 1713759766, 05:22:46 DEBUG "0.6362666666666655" 05:22:46 DEBUG ] 05:22:46 DEBUG } 05:22:46 DEBUG ] 05:22:46 DEBUG } 05:22:46 DEBUG } 05:22:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759767" 05:22:47 INFO [http_cmd]: http status code OK 05:22:47 DEBUG --- status code --- 05:22:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:47 DEBUG --- http response --- 05:22:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:47 DEBUG ----- output ----- 05:22:47 DEBUG { 05:22:47 DEBUG "status": "success", 05:22:47 DEBUG "data": { 05:22:47 DEBUG "resultType": "vector", 05:22:47 DEBUG "result": [ 05:22:47 DEBUG { 05:22:47 DEBUG "metric": { 05:22:47 DEBUG "instance": "10.142.0.107:9100" 05:22:47 DEBUG }, 05:22:47 DEBUG "value": [ 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG [...] 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG "instance": "10.142.0.43:9100" 05:22:47 DEBUG }, 05:22:47 DEBUG "value": [ 05:22:47 DEBUG 1713759767, 05:22:47 DEBUG "5759210339328" 05:22:47 DEBUG ] 05:22:47 DEBUG } 05:22:47 DEBUG ] 05:22:47 DEBUG } 05:22:47 DEBUG } 05:22:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759767" 05:22:47 INFO [http_cmd]: http status code OK 05:22:47 DEBUG --- status code --- 05:22:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:47 DEBUG --- http response --- 05:22:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:47 DEBUG ----- output ----- 05:22:47 DEBUG { 05:22:47 DEBUG "status": "success", 05:22:47 DEBUG "data": { 05:22:47 DEBUG "resultType": "vector", 05:22:47 DEBUG "result": [ 05:22:47 DEBUG { 05:22:47 DEBUG "metric": { 05:22:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:22:47 DEBUG }, 05:22:47 DEBUG "value": [ 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG [...] 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:22:47 DEBUG }, 05:22:47 DEBUG "value": [ 05:22:47 DEBUG 1713759767, 05:22:47 DEBUG "4422.2" 05:22:47 DEBUG ] 05:22:47 DEBUG } 05:22:47 DEBUG ] 05:22:47 DEBUG } 05:22:47 DEBUG } 05:22:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759767" 05:22:47 INFO [http_cmd]: http status code OK 05:22:47 DEBUG --- status code --- 05:22:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:47 DEBUG --- http response --- 05:22:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:22:47 DEBUG ----- output ----- 05:22:47 DEBUG { 05:22:47 DEBUG "status": "success", 05:22:47 DEBUG "data": { 05:22:47 DEBUG "resultType": "vector", 05:22:47 DEBUG "result": [ 05:22:47 DEBUG { 05:22:47 DEBUG "metric": { 05:22:47 DEBUG "pod": "ds-cts-0" 05:22:47 DEBUG }, 05:22:47 DEBUG "value": [ 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG [...] 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG 05:22:47 DEBUG "pod": "ds-cts-2" 05:22:47 DEBUG }, 05:22:47 DEBUG "value": [ 05:22:47 DEBUG 1713759767, 05:22:47 DEBUG "21.9" 05:22:47 DEBUG ] 05:22:47 DEBUG } 05:22:47 DEBUG ] 05:22:47 DEBUG } 05:22:47 DEBUG } 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG { 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "data": { 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "result": [ 05:22:48 DEBUG { 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "pod": "ds-idrepo-0" 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG [...] 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG --- http response --- 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 DEBUG --- status code --- 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG 05:22:48 DEBUG { 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG { 05:22:48 DEBUG 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG 05:22:48 DEBUG "data": { 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG { 05:22:48 DEBUG "data": { 05:22:48 DEBUG "pod": "ds-idrepo-2" 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG { 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "data": { 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG { 05:22:48 DEBUG }, 05:22:48 DEBUG "data": { 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "value": [ 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "result": [ 05:22:48 DEBUG { 05:22:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG "result": [ 05:22:48 DEBUG { 05:22:48 DEBUG "metric": { 05:22:48 DEBUG }, 05:22:48 DEBUG "0" 05:22:48 DEBUG { 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "instance": "10.142.0.107:9100" 05:22:48 DEBUG "value": [ 05:22:48 DEBUG ] 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG } 05:22:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG ] 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG [...] 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG } 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG } 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG "instance": "10.142.0.43:9100" 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:22:48 DEBUG "pod": "ds-cts-2" 05:22:48 DEBUG }, 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG "0.41103333333353853" 05:22:48 DEBUG "value": [ 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG "57577645.411367886" 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG ] 05:22:48 DEBUG "890.2345" 05:22:48 DEBUG ] 05:22:48 DEBUG "599629.5" 05:22:48 DEBUG ] 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG ] 05:22:48 DEBUG ] 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759768" 05:22:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713759768" 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 DEBUG --- status code --- 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG --- http response --- 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 INFO [http_cmd]: http status code OK 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG --- status code --- 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG { 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG { 05:22:48 DEBUG --- http response --- 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:48 DEBUG { 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:48 DEBUG { 05:22:48 DEBUG { 05:22:48 DEBUG { 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG { 05:22:48 DEBUG { 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG "data": { 05:22:48 DEBUG { 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG ----- output ----- 05:22:48 DEBUG "data": { 05:22:48 DEBUG { 05:22:48 DEBUG "data": { 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG { 05:22:48 DEBUG "data": { 05:22:48 DEBUG "data": { 05:22:48 DEBUG "data": { 05:22:48 DEBUG { 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG { 05:22:48 DEBUG { 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "data": { 05:22:48 DEBUG "data": { 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "data": { 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "status": "success", 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "data": { 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "data": { 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "data": { 05:22:48 DEBUG { 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "data": { 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG { 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG { 05:22:48 DEBUG { 05:22:48 DEBUG { 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "instance": "10.142.0.107:9100" 05:22:48 DEBUG { 05:22:48 DEBUG "result": [ 05:22:48 DEBUG { 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "metric": { 05:22:48 DEBUG { 05:22:48 DEBUG { 05:22:48 DEBUG "result": [ 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "result": [ 05:22:48 DEBUG }, 05:22:48 DEBUG "data": { 05:22:48 DEBUG "metric": { 05:22:48 DEBUG { 05:22:48 DEBUG "metric": { 05:22:48 DEBUG { 05:22:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "metric": { 05:22:48 DEBUG { 05:22:48 DEBUG }, 05:22:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:22:48 DEBUG { 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "value": [ 05:22:48 DEBUG "resultType": "vector", 05:22:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "pod": "ds-cts-0" 05:22:48 DEBUG "metric": { 05:22:48 DEBUG }, 05:22:48 DEBUG "instance": "10.142.0.107:9100" 05:22:48 DEBUG "instance": "10.142.0.107:9100" 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "value": [ 05:22:48 DEBUG }, 05:22:48 DEBUG "metric": { 05:22:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:22:48 DEBUG 05:22:48 DEBUG "result": [ 05:22:48 DEBUG }, 05:22:48 DEBUG "pod": "ds-idrepo-0" 05:22:48 DEBUG }, 05:22:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:22:48 DEBUG "value": [ 05:22:48 DEBUG }, 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG { 05:22:48 DEBUG "value": [ 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG "metric": { 05:22:48 DEBUG 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG "pod": "ds-idrepo-0" 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG "value": [ 05:22:48 DEBUG [...] 05:22:48 DEBUG [...] 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG [...] 05:22:48 DEBUG [...] 05:22:48 DEBUG [...] 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG "instance": "10.142.0.43:9100" 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG [...] 05:22:48 DEBUG 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:22:48 DEBUG 05:22:48 DEBUG "pod": "ds-cts-2" 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG "instance": "10.142.0.43:9100" 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG }, 05:22:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:22:48 DEBUG "instance": "10.142.0.43:9100" 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG "3344.366666666667" 05:22:48 DEBUG "pod": "ds-cts-2" 05:22:48 DEBUG 05:22:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:22:48 DEBUG "100730003.6" 05:22:48 DEBUG [...] 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG "value": [ 05:22:48 DEBUG }, 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG "pod": "ds-cts-2" 05:22:48 DEBUG }, 05:22:48 DEBUG ] 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG }, 05:22:48 DEBUG 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG "pod": "ds-cts-2" 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG } 05:22:48 DEBUG 05:22:48 DEBUG 05:22:48 DEBUG ] 05:22:48 DEBUG "45497810.41506646" 05:22:48 DEBUG "value": [ 05:22:48 DEBUG 05:22:48 DEBUG "6627237888" 05:22:48 DEBUG }, 05:22:48 DEBUG "0" 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG "100732663.13333333" 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG ] 05:22:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:22:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG 05:22:48 DEBUG ] 05:22:48 DEBUG "value": [ 05:22:48 DEBUG ] 05:22:48 DEBUG "11.520916600577463" 05:22:48 DEBUG "0.08366666666665878" 05:22:48 DEBUG ] 05:22:48 DEBUG "13.81336126064866" 05:22:48 DEBUG } 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG "3344.4666666666667" 05:22:48 DEBUG "pod": "ds-idrepo-2" 05:22:48 DEBUG } 05:22:48 DEBUG }, 05:22:48 DEBUG "value": [ 05:22:48 DEBUG ] 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG ] 05:22:48 DEBUG }, 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG "0" 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG "value": [ 05:22:48 DEBUG "0" 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG "3339.5666666666666" 05:22:48 DEBUG ] 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG 1713759768, 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG ] 05:22:48 DEBUG "0" 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG ] 05:22:48 DEBUG } 05:22:48 DEBUG } 05:22:48 DEBUG } 05:23:05 INFO 05:23:05 INFO [loop_until]: kubectl --namespace=xlou top pods 05:23:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:23:05 INFO [loop_until]: OK (rc = 0) 05:23:05 DEBUG --- stdout --- 05:23:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22019m 8887Mi am-767f6b67b5-qcqz2 23059m 9007Mi am-767f6b67b5-vbltz 23789m 8912Mi ds-cts-0 20298m 12520Mi ds-cts-1 20366m 12561Mi ds-cts-2 20664m 12572Mi ds-idrepo-0 9630m 3994Mi ds-idrepo-1 9289m 3982Mi ds-idrepo-2 9641m 3964Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 9m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11269m 6239Mi 05:23:05 DEBUG --- stderr --- 05:23:05 DEBUG 05:23:13 INFO 05:23:13 INFO [loop_until]: kubectl --namespace=xlou top node 05:23:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:23:13 INFO [loop_until]: OK (rc = 0) 05:23:13 DEBUG --- stdout --- 05:23:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 6222Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10097m 33% 8126Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22893m 76% 11648Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3340Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 20114m 67% 15792Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5230Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 23122m 77% 11914Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19948m 66% 3247Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25063m 83% 11245Mi 10% gke-xlou-cdm-ds-0f601751-j488 10179m 34% 7482Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20016m 67% 17739Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 9954m 33% 8040Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20224m 67% 15297Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11464m 72% 16159Mi 27% 05:23:13 DEBUG --- stderr --- 05:23:13 DEBUG 05:23:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713759826" 05:23:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759826" 05:23:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759826" 05:23:46 ERROR [http_cmd]: ERROR 05:23:46 INFO [http_cmd]: http status code OK 05:23:46 ERROR --- status code --- 05:23:46 DEBUG --- status code --- 05:23:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:23:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:46 ERROR --- http response --- 05:23:46 DEBUG --- http response --- 05:23:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:23:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:23:46 INFO [http_cmd]: http status code OK 05:23:46 DEBUG --- status code --- 05:23:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:46 DEBUG --- http response --- 05:23:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:46 DEBUG ----- output ----- 05:23:46 DEBUG { 05:23:46 DEBUG "status": "success", 05:23:46 DEBUG "data": { 05:23:46 DEBUG "resultType": "vector", 05:23:46 DEBUG "result": [ 05:23:46 DEBUG { 05:23:46 DEBUG "metric": { 05:23:46 DEBUG "pod": "ds-idrepo-0" 05:23:46 DEBUG }, 05:23:46 DEBUG "value": [ 05:23:46 DEBUG 05:23:46 DEBUG 05:23:46 DEBUG 05:23:46 DEBUG [...] 05:23:46 DEBUG 05:23:46 DEBUG 05:23:46 DEBUG 05:23:46 DEBUG "pod": "ds-idrepo-2" 05:23:46 DEBUG }, 05:23:46 DEBUG "value": [ 05:23:46 DEBUG 1713759826, 05:23:46 DEBUG "0" 05:23:46 DEBUG ] 05:23:46 DEBUG } 05:23:46 DEBUG ] 05:23:46 DEBUG } 05:23:46 DEBUG } 05:23:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759826" 05:23:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759826" 05:23:46 INFO [http_cmd]: http status code OK 05:23:46 DEBUG --- status code --- 05:23:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:46 DEBUG --- http response --- 05:23:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:46 DEBUG ----- output ----- 05:23:46 DEBUG { 05:23:46 DEBUG "status": "success", 05:23:46 DEBUG "data": { 05:23:46 DEBUG "resultType": "vector", 05:23:46 DEBUG "result": [ 05:23:46 DEBUG { 05:23:46 DEBUG "metric": { 05:23:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:23:46 DEBUG }, 05:23:46 DEBUG "value": [ 05:23:46 DEBUG 05:23:46 DEBUG 05:23:46 DEBUG 05:23:46 INFO [http_cmd]: http status code OK 05:23:46 DEBUG [...] 05:23:46 DEBUG --- status code --- 05:23:46 DEBUG 05:23:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:46 DEBUG 05:23:46 DEBUG --- http response --- 05:23:46 DEBUG 05:23:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:23:46 DEBUG ----- output ----- 05:23:46 DEBUG { 05:23:46 DEBUG "status": "success", 05:23:46 DEBUG "data": { 05:23:46 DEBUG "resultType": "vector", 05:23:46 DEBUG "result": [ 05:23:46 DEBUG { 05:23:46 DEBUG "metric": { 05:23:46 DEBUG "pod": "ds-cts-0" 05:23:46 DEBUG }, 05:23:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:46 DEBUG "value": [ 05:23:46 DEBUG }, 05:23:46 DEBUG 05:23:46 DEBUG 05:23:46 DEBUG "value": [ 05:23:46 DEBUG 05:23:46 DEBUG [...] 05:23:46 DEBUG 05:23:46 DEBUG 05:23:46 DEBUG 05:23:46 DEBUG 1713759826, 05:23:46 DEBUG "pod": "ds-cts-2" 05:23:46 DEBUG "492693.4378060725" 05:23:46 DEBUG }, 05:23:46 DEBUG ] 05:23:46 DEBUG "value": [ 05:23:46 DEBUG } 05:23:46 DEBUG 1713759826, 05:23:46 DEBUG ] 05:23:46 DEBUG "0.615633333333335" 05:23:46 DEBUG } 05:23:46 DEBUG ] 05:23:46 DEBUG } 05:23:46 DEBUG } 05:23:46 DEBUG ] 05:23:46 DEBUG } 05:23:46 DEBUG } 05:23:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759827" 05:23:47 INFO [http_cmd]: http status code OK 05:23:47 DEBUG --- status code --- 05:23:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:47 DEBUG --- http response --- 05:23:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:47 DEBUG ----- output ----- 05:23:47 DEBUG { 05:23:47 DEBUG "status": "success", 05:23:47 DEBUG "data": { 05:23:47 DEBUG "resultType": "vector", 05:23:47 DEBUG "result": [ 05:23:47 DEBUG { 05:23:47 DEBUG "metric": { 05:23:47 DEBUG "instance": "10.142.0.107:9100" 05:23:47 DEBUG }, 05:23:47 DEBUG "value": [ 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG [...] 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG "instance": "10.142.0.43:9100" 05:23:47 DEBUG }, 05:23:47 DEBUG "value": [ 05:23:47 DEBUG 1713759827, 05:23:47 DEBUG "5764356649984" 05:23:47 DEBUG ] 05:23:47 DEBUG } 05:23:47 DEBUG ] 05:23:47 DEBUG } 05:23:47 DEBUG } 05:23:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759827" 05:23:47 INFO [http_cmd]: http status code OK 05:23:47 DEBUG --- status code --- 05:23:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:47 DEBUG --- http response --- 05:23:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:47 DEBUG ----- output ----- 05:23:47 DEBUG { 05:23:47 DEBUG "status": "success", 05:23:47 DEBUG "data": { 05:23:47 DEBUG "resultType": "vector", 05:23:47 DEBUG "result": [ 05:23:47 DEBUG { 05:23:47 DEBUG "metric": { 05:23:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:23:47 DEBUG }, 05:23:47 DEBUG "value": [ 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG [...] 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:47 DEBUG }, 05:23:47 DEBUG "value": [ 05:23:47 DEBUG 1713759827, 05:23:47 DEBUG "4340.166666666666" 05:23:47 DEBUG ] 05:23:47 DEBUG } 05:23:47 DEBUG ] 05:23:47 DEBUG } 05:23:47 DEBUG } 05:23:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759827" 05:23:47 INFO [http_cmd]: http status code OK 05:23:47 DEBUG --- status code --- 05:23:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:47 DEBUG --- http response --- 05:23:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:23:47 DEBUG ----- output ----- 05:23:47 DEBUG { 05:23:47 DEBUG "status": "success", 05:23:47 DEBUG "data": { 05:23:47 DEBUG "resultType": "vector", 05:23:47 DEBUG "result": [ 05:23:47 DEBUG { 05:23:47 DEBUG "metric": { 05:23:47 DEBUG "pod": "ds-cts-0" 05:23:47 DEBUG }, 05:23:47 DEBUG "value": [ 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG [...] 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG 05:23:47 DEBUG "pod": "ds-cts-2" 05:23:47 DEBUG }, 05:23:47 DEBUG "value": [ 05:23:47 DEBUG 1713759827, 05:23:47 DEBUG "21.1" 05:23:47 DEBUG ] 05:23:47 DEBUG } 05:23:47 DEBUG ] 05:23:47 DEBUG } 05:23:47 DEBUG } 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:48 DEBUG { 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG "data": { 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "result": [ 05:23:48 DEBUG { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG { 05:23:48 DEBUG "pod": "ds-idrepo-0" 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 DEBUG "status": "success", 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 DEBUG }, 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG "value": [ 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG 05:23:48 DEBUG "data": { 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:48 DEBUG 05:23:48 DEBUG "result": [ 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 DEBUG { 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG [...] 05:23:48 DEBUG { 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG { 05:23:48 DEBUG 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG 05:23:48 DEBUG "data": { 05:23:48 DEBUG }, 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG "data": { 05:23:48 DEBUG 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:48 DEBUG "pod": "ds-idrepo-2" 05:23:48 DEBUG "result": [ 05:23:48 DEBUG 05:23:48 DEBUG "result": [ 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG }, 05:23:48 DEBUG { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:23:48 DEBUG 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG }, 05:23:48 DEBUG 05:23:48 DEBUG [...] 05:23:48 DEBUG "data": { 05:23:48 DEBUG "0" 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG ] 05:23:48 DEBUG 05:23:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:23:48 DEBUG 05:23:48 DEBUG "result": [ 05:23:48 DEBUG } 05:23:48 DEBUG 05:23:48 DEBUG }, 05:23:48 DEBUG 05:23:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:48 DEBUG ] 05:23:48 DEBUG 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG }, 05:23:48 DEBUG } 05:23:48 DEBUG 05:23:48 DEBUG "value": [ 05:23:48 DEBUG { 05:23:48 DEBUG "value": [ 05:23:48 DEBUG } 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "metric": { 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:48 DEBUG }, 05:23:48 DEBUG "instance": "10.142.0.107:9100" 05:23:48 DEBUG "597106" 05:23:48 DEBUG "value": [ 05:23:48 DEBUG }, 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "901.517" 05:23:48 DEBUG 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG [...] 05:23:48 DEBUG ] 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG } 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG "instance": "10.142.0.43:9100" 05:23:48 DEBUG 05:23:48 DEBUG } 05:23:48 DEBUG }, 05:23:48 DEBUG 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:48 DEBUG "0.3706333333332983" 05:23:48 DEBUG }, 05:23:48 DEBUG ] 05:23:48 DEBUG "value": [ 05:23:48 DEBUG } 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG "3008.8148873653286" 05:23:48 DEBUG ] 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759828" 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 DEBUG --- status code --- 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG --- status code --- 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 DEBUG --- status code --- 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG --- http response --- 05:23:48 INFO [http_cmd]: http status code OK 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG --- status code --- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:23:48 DEBUG --- http response --- 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG { 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:48 DEBUG { 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG { 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG ----- output ----- 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG "data": { 05:23:48 DEBUG "data": { 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG "data": { 05:23:48 DEBUG "data": { 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG "data": { 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG { 05:23:48 DEBUG "data": { 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "data": { 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "data": { 05:23:48 DEBUG "data": { 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "data": { 05:23:48 DEBUG "data": { 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "status": "success", 05:23:48 DEBUG "data": { 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "result": [ 05:23:48 DEBUG { 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "data": { 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG "data": { 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG { 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "result": [ 05:23:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:23:48 DEBUG "resultType": "vector", 05:23:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:23:48 DEBUG "metric": { 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG { 05:23:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:23:48 DEBUG "instance": "10.142.0.107:9100" 05:23:48 DEBUG "instance": "10.142.0.107:9100" 05:23:48 DEBUG "result": [ 05:23:48 DEBUG { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "result": [ 05:23:48 DEBUG }, 05:23:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG "metric": { 05:23:48 DEBUG }, 05:23:48 DEBUG }, 05:23:48 DEBUG }, 05:23:48 DEBUG { 05:23:48 DEBUG "pod": "ds-idrepo-0" 05:23:48 DEBUG }, 05:23:48 DEBUG { 05:23:48 DEBUG "value": [ 05:23:48 DEBUG }, 05:23:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:23:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:23:48 DEBUG "pod": "ds-cts-0" 05:23:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:23:48 DEBUG }, 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "metric": { 05:23:48 DEBUG }, 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "metric": { 05:23:48 DEBUG 05:23:48 DEBUG "value": [ 05:23:48 DEBUG }, 05:23:48 DEBUG }, 05:23:48 DEBUG "value": [ 05:23:48 DEBUG }, 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "pod": "ds-idrepo-0" 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 05:23:48 DEBUG "instance": "10.142.0.107:9100" 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG }, 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG }, 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "value": [ 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG }, 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG [...] 05:23:48 DEBUG [...] 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG [...] 05:23:48 DEBUG "value": [ 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG [...] 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:48 DEBUG 05:23:48 DEBUG }, 05:23:48 DEBUG 05:23:48 DEBUG "instance": "10.142.0.43:9100" 05:23:48 DEBUG 05:23:48 DEBUG "pod": "ds-cts-2" 05:23:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:48 DEBUG }, 05:23:48 DEBUG }, 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:48 DEBUG }, 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 05:23:48 DEBUG "instance": "10.142.0.43:9100" 05:23:48 DEBUG }, 05:23:48 DEBUG 05:23:48 DEBUG }, 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "value": [ 05:23:48 DEBUG [...] 05:23:48 DEBUG 05:23:48 DEBUG 05:23:48 DEBUG }, 05:23:48 DEBUG 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG "pod": "ds-cts-2" 05:23:48 DEBUG }, 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "pod": "ds-idrepo-2" 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG "0" 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 05:23:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:23:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:23:48 DEBUG }, 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG "value": [ 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG }, 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG 05:23:48 DEBUG }, 05:23:48 DEBUG }, 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG "24.532983104801154" 05:23:48 DEBUG ] 05:23:48 DEBUG "3326.366666666667" 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG "102153479.9" 05:23:48 DEBUG "value": [ 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG "0" 05:23:48 DEBUG 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "value": [ 05:23:48 DEBUG "0" 05:23:48 DEBUG "11.61064030500191" 05:23:48 DEBUG }, 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG "102150080.46666667" 05:23:48 DEBUG ] 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG } 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG "value": [ 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG "0" 05:23:48 DEBUG "3326.2666666666664" 05:23:48 DEBUG ] 05:23:48 DEBUG "instance": "10.142.0.43:9100" 05:23:48 DEBUG "6633574400" 05:23:48 DEBUG "3326.2999999999997" 05:23:48 DEBUG ] 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG ] 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG }, 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG "0" 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG "value": [ 05:23:48 DEBUG ] 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG 1713759828, 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG "0.07333333333333106" 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG } 05:23:48 DEBUG ] 05:23:48 DEBUG } 05:23:48 DEBUG } 05:24:05 INFO 05:24:05 INFO [loop_until]: kubectl --namespace=xlou top pods 05:24:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:24:05 INFO [loop_until]: OK (rc = 0) 05:24:05 DEBUG --- stdout --- 05:24:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22750m 8887Mi am-767f6b67b5-qcqz2 23280m 8970Mi am-767f6b67b5-vbltz 25079m 8922Mi ds-cts-0 20153m 12523Mi ds-cts-1 20063m 12556Mi ds-cts-2 20288m 12566Mi ds-idrepo-0 9711m 3994Mi ds-idrepo-1 9795m 3981Mi ds-idrepo-2 9667m 3965Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 11m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11484m 6235Mi 05:24:05 DEBUG --- stderr --- 05:24:05 DEBUG 05:24:13 INFO 05:24:13 INFO [loop_until]: kubectl --namespace=xlou top node 05:24:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:24:13 INFO [loop_until]: OK (rc = 0) 05:24:13 DEBUG --- stdout --- 05:24:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6224Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10003m 33% 8116Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22530m 75% 11646Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3347Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 20250m 67% 15796Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5211Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 23200m 77% 11893Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19346m 64% 3243Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25334m 84% 11245Mi 10% gke-xlou-cdm-ds-0f601751-j488 10229m 34% 7479Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19849m 66% 17743Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10026m 33% 8047Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20326m 68% 15302Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11828m 74% 16161Mi 27% 05:24:13 DEBUG --- stderr --- 05:24:13 DEBUG 05:24:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759886" 05:24:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713759886" 05:24:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759886" 05:24:46 ERROR [http_cmd]: ERROR 05:24:46 INFO [http_cmd]: http status code OK 05:24:46 INFO [http_cmd]: http status code OK 05:24:46 ERROR --- status code --- 05:24:46 DEBUG --- status code --- 05:24:46 DEBUG --- status code --- 05:24:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:24:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:46 ERROR --- http response --- 05:24:46 DEBUG --- http response --- 05:24:46 DEBUG --- http response --- 05:24:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:24:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:24:46 DEBUG ----- output ----- 05:24:46 DEBUG { 05:24:46 DEBUG "status": "success", 05:24:46 DEBUG "data": { 05:24:46 DEBUG "resultType": "vector", 05:24:46 DEBUG "result": [ 05:24:46 DEBUG { 05:24:46 DEBUG "metric": { 05:24:46 DEBUG "pod": "ds-idrepo-0" 05:24:46 DEBUG }, 05:24:46 DEBUG "value": [ 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG [...] 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG "pod": "ds-idrepo-2" 05:24:46 DEBUG }, 05:24:46 DEBUG "value": [ 05:24:46 DEBUG 1713759886, 05:24:46 DEBUG "0" 05:24:46 DEBUG ] 05:24:46 DEBUG } 05:24:46 DEBUG ] 05:24:46 DEBUG } 05:24:46 DEBUG } 05:24:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759886" 05:24:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759886" 05:24:46 INFO [http_cmd]: http status code OK 05:24:46 INFO [http_cmd]: http status code OK 05:24:46 DEBUG --- status code --- 05:24:46 DEBUG --- status code --- 05:24:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:46 DEBUG --- http response --- 05:24:46 DEBUG --- http response --- 05:24:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:24:46 DEBUG ----- output ----- 05:24:46 DEBUG { 05:24:46 DEBUG "status": "success", 05:24:46 DEBUG "data": { 05:24:46 DEBUG "resultType": "vector", 05:24:46 DEBUG "result": [ 05:24:46 DEBUG { 05:24:46 DEBUG "metric": { 05:24:46 DEBUG "pod": "ds-cts-0" 05:24:46 DEBUG }, 05:24:46 DEBUG "value": [ 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG [...] 05:24:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:24:46 DEBUG ----- output ----- 05:24:46 DEBUG { 05:24:46 DEBUG "status": "success", 05:24:46 DEBUG "data": { 05:24:46 DEBUG "resultType": "vector", 05:24:46 DEBUG "result": [ 05:24:46 DEBUG { 05:24:46 DEBUG "metric": { 05:24:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:24:46 DEBUG }, 05:24:46 DEBUG "value": [ 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG 05:24:46 DEBUG "pod": "ds-cts-2" 05:24:46 DEBUG [...] 05:24:46 DEBUG 05:24:46 DEBUG }, 05:24:46 DEBUG 05:24:46 DEBUG "value": [ 05:24:46 DEBUG 05:24:46 DEBUG 1713759886, 05:24:46 DEBUG "pod": "ds-cts-2" 05:24:46 DEBUG "0.6015666666666599" 05:24:46 DEBUG }, 05:24:46 DEBUG ] 05:24:46 DEBUG "value": [ 05:24:46 DEBUG } 05:24:46 DEBUG 1713759886, 05:24:46 DEBUG ] 05:24:46 DEBUG "350373.26353258966" 05:24:46 DEBUG } 05:24:46 DEBUG ] 05:24:46 DEBUG } 05:24:46 DEBUG } 05:24:46 DEBUG ] 05:24:46 DEBUG } 05:24:46 DEBUG } 05:24:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759887" 05:24:47 INFO [http_cmd]: http status code OK 05:24:47 DEBUG --- status code --- 05:24:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:47 DEBUG --- http response --- 05:24:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:47 DEBUG ----- output ----- 05:24:47 DEBUG { 05:24:47 DEBUG "status": "success", 05:24:47 DEBUG "data": { 05:24:47 DEBUG "resultType": "vector", 05:24:47 DEBUG "result": [ 05:24:47 DEBUG { 05:24:47 DEBUG "metric": { 05:24:47 DEBUG "instance": "10.142.0.107:9100" 05:24:47 DEBUG }, 05:24:47 DEBUG "value": [ 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG [...] 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG "instance": "10.142.0.43:9100" 05:24:47 DEBUG }, 05:24:47 DEBUG "value": [ 05:24:47 DEBUG 1713759887, 05:24:47 DEBUG "5769363010560" 05:24:47 DEBUG ] 05:24:47 DEBUG } 05:24:47 DEBUG ] 05:24:47 DEBUG } 05:24:47 DEBUG } 05:24:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759887" 05:24:47 INFO [http_cmd]: http status code OK 05:24:47 DEBUG --- status code --- 05:24:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:47 DEBUG --- http response --- 05:24:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:47 DEBUG ----- output ----- 05:24:47 DEBUG { 05:24:47 DEBUG "status": "success", 05:24:47 DEBUG "data": { 05:24:47 DEBUG "resultType": "vector", 05:24:47 DEBUG "result": [ 05:24:47 DEBUG { 05:24:47 DEBUG "metric": { 05:24:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:24:47 DEBUG }, 05:24:47 DEBUG "value": [ 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG [...] 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:24:47 DEBUG }, 05:24:47 DEBUG "value": [ 05:24:47 DEBUG 1713759887, 05:24:47 DEBUG "4331.5" 05:24:47 DEBUG ] 05:24:47 DEBUG } 05:24:47 DEBUG ] 05:24:47 DEBUG } 05:24:47 DEBUG } 05:24:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759887" 05:24:47 INFO [http_cmd]: http status code OK 05:24:47 DEBUG --- status code --- 05:24:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:47 DEBUG --- http response --- 05:24:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:24:47 DEBUG ----- output ----- 05:24:47 DEBUG { 05:24:47 DEBUG "status": "success", 05:24:47 DEBUG "data": { 05:24:47 DEBUG "resultType": "vector", 05:24:47 DEBUG "result": [ 05:24:47 DEBUG { 05:24:47 DEBUG "metric": { 05:24:47 DEBUG "pod": "ds-cts-0" 05:24:47 DEBUG }, 05:24:47 DEBUG "value": [ 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG [...] 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG 05:24:47 DEBUG "pod": "ds-cts-2" 05:24:47 DEBUG }, 05:24:47 DEBUG "value": [ 05:24:47 DEBUG 1713759887, 05:24:47 DEBUG "21.766666666666666" 05:24:47 DEBUG ] 05:24:47 DEBUG } 05:24:47 DEBUG ] 05:24:47 DEBUG } 05:24:47 DEBUG } 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG { 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG "data": { 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG "result": [ 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:48 DEBUG { 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG "metric": { 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG { 05:24:48 DEBUG "pod": "ds-idrepo-0" 05:24:48 DEBUG { 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG }, 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "data": { 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:48 DEBUG "value": [ 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG "data": { 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG "result": [ 05:24:48 DEBUG { 05:24:48 DEBUG 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG "result": [ 05:24:48 DEBUG { 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG "metric": { 05:24:48 DEBUG "data": { 05:24:48 DEBUG [...] 05:24:48 DEBUG { 05:24:48 DEBUG { 05:24:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "metric": { 05:24:48 DEBUG }, 05:24:48 DEBUG "result": [ 05:24:48 DEBUG 05:24:48 DEBUG "data": { 05:24:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "result": [ 05:24:48 DEBUG { 05:24:48 DEBUG "value": [ 05:24:48 DEBUG [...] 05:24:48 DEBUG "pod": "ds-idrepo-2" 05:24:48 DEBUG { 05:24:48 DEBUG "metric": { 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG }, 05:24:48 DEBUG "metric": { 05:24:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "instance": "10.142.0.107:9100" 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:24:48 DEBUG "0" 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG }, 05:24:48 DEBUG ] 05:24:48 DEBUG 05:24:48 DEBUG [...] 05:24:48 DEBUG 05:24:48 DEBUG [...] 05:24:48 DEBUG } 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG ] 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG } 05:24:48 DEBUG [...] 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:24:48 DEBUG 05:24:48 DEBUG } 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG }, 05:24:48 DEBUG "911.697" 05:24:48 DEBUG "instance": "10.142.0.43:9100" 05:24:48 DEBUG "pod": "ds-cts-2" 05:24:48 DEBUG "value": [ 05:24:48 DEBUG ] 05:24:48 DEBUG }, 05:24:48 DEBUG }, 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG } 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "597889" 05:24:48 DEBUG ] 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG "0.3801666666664914" 05:24:48 DEBUG "34284023.29852917" 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759888" 05:24:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759888" 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG --- status code --- 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG --- status code --- 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG --- status code --- 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG --- http response --- 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:48 DEBUG { 05:24:48 INFO [http_cmd]: http status code OK 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG { 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG { 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG --- status code --- 05:24:48 DEBUG { 05:24:48 DEBUG { 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG { 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG { 05:24:48 DEBUG { 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG { 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG { 05:24:48 DEBUG "data": { 05:24:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG { 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "data": { 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "data": { 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG --- http response --- 05:24:48 DEBUG "data": { 05:24:48 DEBUG "data": { 05:24:48 DEBUG { 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "data": { 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG "data": { 05:24:48 DEBUG "data": { 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG "data": { 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG "data": { 05:24:48 DEBUG "result": [ 05:24:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG "data": { 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG { 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG { 05:24:48 DEBUG ----- output ----- 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG "data": { 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "result": [ 05:24:48 DEBUG { 05:24:48 DEBUG "result": [ 05:24:48 DEBUG { 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "metric": { 05:24:48 DEBUG { 05:24:48 DEBUG { 05:24:48 DEBUG { 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG { 05:24:48 DEBUG "data": { 05:24:48 DEBUG { 05:24:48 DEBUG { 05:24:48 DEBUG "metric": { 05:24:48 DEBUG { 05:24:48 DEBUG "metric": { 05:24:48 DEBUG { 05:24:48 DEBUG "instance": "10.142.0.107:9100" 05:24:48 DEBUG "status": "success", 05:24:48 DEBUG "metric": { 05:24:48 DEBUG "metric": { 05:24:48 DEBUG { 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "metric": { 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG "metric": { 05:24:48 DEBUG "metric": { 05:24:48 DEBUG "pod": "ds-idrepo-0" 05:24:48 DEBUG "metric": { 05:24:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:24:48 DEBUG "metric": { 05:24:48 DEBUG }, 05:24:48 DEBUG "data": { 05:24:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:24:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:24:48 DEBUG }, 05:24:48 DEBUG { 05:24:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:24:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:24:48 DEBUG }, 05:24:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:24:48 DEBUG }, 05:24:48 DEBUG "instance": "10.142.0.107:9100" 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "resultType": "vector", 05:24:48 DEBUG }, 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "metric": { 05:24:48 DEBUG }, 05:24:48 DEBUG { 05:24:48 DEBUG }, 05:24:48 DEBUG }, 05:24:48 DEBUG "value": [ 05:24:48 DEBUG }, 05:24:48 DEBUG "value": [ 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG "result": [ 05:24:48 DEBUG "metric": { 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "pod": "ds-idrepo-0" 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "metric": { 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG { 05:24:48 DEBUG "pod": "ds-cts-0" 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "metric": { 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG [...] 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG [...] 05:24:48 DEBUG [...] 05:24:48 DEBUG "value": [ 05:24:48 DEBUG [...] 05:24:48 DEBUG [...] 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "instance": "10.142.0.107:9100" 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG [...] 05:24:48 DEBUG [...] 05:24:48 DEBUG [...] 05:24:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG [...] 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG [...] 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG [...] 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG [...] 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "instance": "10.142.0.43:9100" 05:24:48 DEBUG "pod": "ds-cts-2" 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "pod": "ds-cts-2" 05:24:48 DEBUG }, 05:24:48 DEBUG }, 05:24:48 DEBUG "pod": "ds-cts-2" 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG [...] 05:24:48 DEBUG 05:24:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:24:48 DEBUG }, 05:24:48 DEBUG "instance": "10.142.0.43:9100" 05:24:48 DEBUG "6632157184" 05:24:48 DEBUG 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "value": [ 05:24:48 DEBUG }, 05:24:48 DEBUG 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG 05:24:48 DEBUG 05:24:48 DEBUG "pod": "ds-idrepo-2" 05:24:48 DEBUG "value": [ 05:24:48 DEBUG }, 05:24:48 DEBUG ] 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "pod": "ds-cts-2" 05:24:48 DEBUG "11.62097825717225" 05:24:48 DEBUG 05:24:48 DEBUG }, 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:24:48 DEBUG "value": [ 05:24:48 DEBUG } 05:24:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG 05:24:48 DEBUG "101673499.93333334" 05:24:48 DEBUG "0" 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG "24012306.551012933" 05:24:48 DEBUG ] 05:24:48 DEBUG 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "0" 05:24:48 DEBUG }, 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG ] 05:24:48 DEBUG }, 05:24:48 DEBUG "9.571951953582971" 05:24:48 DEBUG [...] 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG }, 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG ] 05:24:48 DEBUG "value": [ 05:24:48 DEBUG "101675022.63333333" 05:24:48 DEBUG } 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:24:48 DEBUG "0" 05:24:48 DEBUG } 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG "3333.1666666666665" 05:24:48 DEBUG 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG "0" 05:24:48 DEBUG }, 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG "3332.366666666667" 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG "value": [ 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG "instance": "10.142.0.43:9100" 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG "3333.3333333333335" 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG }, 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG "value": [ 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG 1713759888, 05:24:48 DEBUG } 05:24:48 DEBUG "0.07266666666664984" 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG ] 05:24:48 DEBUG } 05:24:48 DEBUG } 05:25:05 INFO 05:25:05 INFO [loop_until]: kubectl --namespace=xlou top pods 05:25:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:25:06 INFO [loop_until]: OK (rc = 0) 05:25:06 DEBUG --- stdout --- 05:25:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 23043m 8890Mi am-767f6b67b5-qcqz2 23085m 8956Mi am-767f6b67b5-vbltz 24975m 8918Mi ds-cts-0 19962m 12515Mi ds-cts-1 19840m 12560Mi ds-cts-2 20095m 12569Mi ds-idrepo-0 9571m 3994Mi ds-idrepo-1 9304m 3981Mi ds-idrepo-2 9693m 3965Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1181Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 9m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11680m 6239Mi 05:25:06 DEBUG --- stderr --- 05:25:06 DEBUG 05:25:13 INFO 05:25:13 INFO [loop_until]: kubectl --namespace=xlou top node 05:25:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:25:13 INFO [loop_until]: OK (rc = 0) 05:25:13 DEBUG --- stdout --- 05:25:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6222Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9960m 33% 8125Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22093m 74% 11650Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3348Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 20223m 67% 15788Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5208Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 23129m 77% 11875Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 20090m 67% 3281Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25184m 84% 11246Mi 10% gke-xlou-cdm-ds-0f601751-j488 10172m 34% 7482Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20059m 67% 17741Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10054m 33% 8040Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20315m 68% 15289Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11873m 74% 16161Mi 27% 05:25:13 DEBUG --- stderr --- 05:25:13 DEBUG 05:25:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713759946" 05:25:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759946" 05:25:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759946" 05:25:46 ERROR [http_cmd]: ERROR 05:25:46 INFO [http_cmd]: http status code OK 05:25:46 INFO [http_cmd]: http status code OK 05:25:46 ERROR --- status code --- 05:25:46 DEBUG --- status code --- 05:25:46 DEBUG --- status code --- 05:25:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:25:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:46 ERROR --- http response --- 05:25:46 DEBUG --- http response --- 05:25:46 DEBUG --- http response --- 05:25:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:25:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:25:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:46 DEBUG ----- output ----- 05:25:46 DEBUG { 05:25:46 DEBUG "status": "success", 05:25:46 DEBUG "data": { 05:25:46 DEBUG "resultType": "vector", 05:25:46 DEBUG "result": [ 05:25:46 DEBUG { 05:25:46 DEBUG "metric": { 05:25:46 DEBUG "pod": "ds-idrepo-0" 05:25:46 DEBUG }, 05:25:46 DEBUG "value": [ 05:25:46 DEBUG 05:25:46 DEBUG 05:25:46 DEBUG 05:25:46 DEBUG [...] 05:25:46 DEBUG 05:25:46 DEBUG 05:25:46 DEBUG 05:25:46 DEBUG "pod": "ds-idrepo-2" 05:25:46 DEBUG }, 05:25:46 DEBUG "value": [ 05:25:46 DEBUG 1713759946, 05:25:46 DEBUG "0" 05:25:46 DEBUG ] 05:25:46 DEBUG } 05:25:46 DEBUG ] 05:25:46 DEBUG } 05:25:46 DEBUG } 05:25:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759946" 05:25:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759946" 05:25:46 INFO [http_cmd]: http status code OK 05:25:46 INFO [http_cmd]: http status code OK 05:25:46 DEBUG --- status code --- 05:25:46 DEBUG --- status code --- 05:25:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:46 DEBUG --- http response --- 05:25:46 DEBUG --- http response --- 05:25:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:25:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:25:46 DEBUG ----- output ----- 05:25:46 DEBUG ----- output ----- 05:25:46 DEBUG { 05:25:46 DEBUG { 05:25:46 DEBUG "status": "success", 05:25:46 DEBUG "status": "success", 05:25:46 DEBUG "data": { 05:25:46 DEBUG "data": { 05:25:46 DEBUG "resultType": "vector", 05:25:46 DEBUG "result": [ 05:25:46 DEBUG { 05:25:46 DEBUG "metric": { 05:25:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:25:46 DEBUG }, 05:25:46 DEBUG "value": [ 05:25:46 DEBUG 05:25:46 DEBUG 05:25:46 DEBUG 05:25:46 DEBUG "resultType": "vector", 05:25:46 DEBUG [...] 05:25:46 DEBUG "result": [ 05:25:46 DEBUG 05:25:46 DEBUG { 05:25:46 DEBUG 05:25:46 DEBUG "metric": { 05:25:46 DEBUG 05:25:46 DEBUG "pod": "ds-cts-0" 05:25:46 DEBUG "pod": "ds-cts-2" 05:25:46 DEBUG }, 05:25:46 DEBUG }, 05:25:46 DEBUG "value": [ 05:25:46 DEBUG 05:25:46 DEBUG "value": [ 05:25:46 DEBUG 05:25:46 DEBUG 1713759946, 05:25:46 DEBUG 05:25:46 DEBUG "0" 05:25:46 DEBUG [...] 05:25:46 DEBUG ] 05:25:46 DEBUG 05:25:46 DEBUG } 05:25:46 DEBUG 05:25:46 DEBUG ] 05:25:46 DEBUG 05:25:46 DEBUG } 05:25:46 DEBUG "pod": "ds-cts-2" 05:25:46 DEBUG } 05:25:46 DEBUG }, 05:25:46 DEBUG "value": [ 05:25:46 DEBUG 1713759946, 05:25:46 DEBUG "0.6005999999999858" 05:25:46 DEBUG ] 05:25:46 DEBUG } 05:25:46 DEBUG ] 05:25:46 DEBUG } 05:25:46 DEBUG } 05:25:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759947" 05:25:47 INFO [http_cmd]: http status code OK 05:25:47 DEBUG --- status code --- 05:25:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:47 DEBUG --- http response --- 05:25:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:47 DEBUG ----- output ----- 05:25:47 DEBUG { 05:25:47 DEBUG "status": "success", 05:25:47 DEBUG "data": { 05:25:47 DEBUG "resultType": "vector", 05:25:47 DEBUG "result": [ 05:25:47 DEBUG { 05:25:47 DEBUG "metric": { 05:25:47 DEBUG "instance": "10.142.0.107:9100" 05:25:47 DEBUG }, 05:25:47 DEBUG "value": [ 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG [...] 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG "instance": "10.142.0.43:9100" 05:25:47 DEBUG }, 05:25:47 DEBUG "value": [ 05:25:47 DEBUG 1713759947, 05:25:47 DEBUG "5774365017088" 05:25:47 DEBUG ] 05:25:47 DEBUG } 05:25:47 DEBUG ] 05:25:47 DEBUG } 05:25:47 DEBUG } 05:25:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759947" 05:25:47 INFO [http_cmd]: http status code OK 05:25:47 DEBUG --- status code --- 05:25:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:47 DEBUG --- http response --- 05:25:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:47 DEBUG ----- output ----- 05:25:47 DEBUG { 05:25:47 DEBUG "status": "success", 05:25:47 DEBUG "data": { 05:25:47 DEBUG "resultType": "vector", 05:25:47 DEBUG "result": [ 05:25:47 DEBUG { 05:25:47 DEBUG "metric": { 05:25:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:25:47 DEBUG }, 05:25:47 DEBUG "value": [ 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG [...] 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:25:47 DEBUG }, 05:25:47 DEBUG "value": [ 05:25:47 DEBUG 1713759947, 05:25:47 DEBUG "4232.833333333333" 05:25:47 DEBUG ] 05:25:47 DEBUG } 05:25:47 DEBUG ] 05:25:47 DEBUG } 05:25:47 DEBUG } 05:25:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759947" 05:25:47 INFO [http_cmd]: http status code OK 05:25:47 DEBUG --- status code --- 05:25:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:47 DEBUG --- http response --- 05:25:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:25:47 DEBUG ----- output ----- 05:25:47 DEBUG { 05:25:47 DEBUG "status": "success", 05:25:47 DEBUG "data": { 05:25:47 DEBUG "resultType": "vector", 05:25:47 DEBUG "result": [ 05:25:47 DEBUG { 05:25:47 DEBUG "metric": { 05:25:47 DEBUG "pod": "ds-cts-0" 05:25:47 DEBUG }, 05:25:47 DEBUG "value": [ 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG [...] 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG 05:25:47 DEBUG "pod": "ds-cts-2" 05:25:47 DEBUG }, 05:25:47 DEBUG "value": [ 05:25:47 DEBUG 1713759947, 05:25:47 DEBUG "21.6" 05:25:47 DEBUG ] 05:25:47 DEBUG } 05:25:47 DEBUG ] 05:25:47 DEBUG } 05:25:47 DEBUG } 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: http status code OK 05:25:48 DEBUG --- status code --- 05:25:48 INFO [http_cmd]: http status code OK 05:25:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:48 DEBUG --- status code --- 05:25:48 DEBUG --- http response --- 05:25:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:48 DEBUG --- http response --- 05:25:48 DEBUG ----- output ----- 05:25:48 INFO [http_cmd]: http status code OK 05:25:48 DEBUG { 05:25:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:48 DEBUG --- status code --- 05:25:48 DEBUG "status": "success", 05:25:48 DEBUG ----- output ----- 05:25:48 INFO [http_cmd]: http status code OK 05:25:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:48 DEBUG "data": { 05:25:48 DEBUG { 05:25:48 DEBUG --- status code --- 05:25:48 DEBUG --- http response --- 05:25:48 DEBUG "resultType": "vector", 05:25:48 DEBUG "status": "success", 05:25:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:48 INFO [http_cmd]: http status code OK 05:25:48 DEBUG "result": [ 05:25:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:48 DEBUG "data": { 05:25:48 DEBUG --- http response --- 05:25:48 DEBUG --- status code --- 05:25:48 DEBUG { 05:25:48 DEBUG ----- output ----- 05:25:48 DEBUG "resultType": "vector", 05:25:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:48 DEBUG "metric": { 05:25:48 DEBUG "pod": "ds-idrepo-0" 05:25:48 DEBUG "result": [ 05:25:48 DEBUG ----- output ----- 05:25:48 DEBUG --- http response --- 05:25:48 DEBUG { 05:25:48 DEBUG }, 05:25:48 DEBUG { 05:25:48 DEBUG { 05:25:48 DEBUG "status": "success", 05:25:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:48 DEBUG "value": [ 05:25:48 DEBUG "metric": { 05:25:48 DEBUG "status": "success", 05:25:48 DEBUG "data": { 05:25:48 DEBUG ----- output ----- 05:25:48 DEBUG 05:25:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:25:48 DEBUG "data": { 05:25:48 DEBUG "resultType": "vector", 05:25:48 DEBUG { 05:25:48 DEBUG 05:25:48 DEBUG }, 05:25:48 DEBUG "resultType": "vector", 05:25:48 DEBUG "result": [ 05:25:48 DEBUG "status": "success", 05:25:48 DEBUG 05:25:48 DEBUG "value": [ 05:25:48 DEBUG 05:25:48 DEBUG { 05:25:48 DEBUG "data": { 05:25:48 DEBUG [...] 05:25:48 DEBUG 05:25:48 DEBUG "result": [ 05:25:48 DEBUG "metric": { 05:25:48 DEBUG "resultType": "vector", 05:25:48 DEBUG 05:25:48 DEBUG 05:25:48 DEBUG { 05:25:48 DEBUG "instance": "10.142.0.107:9100" 05:25:48 DEBUG "result": [ 05:25:48 DEBUG 05:25:48 DEBUG [...] 05:25:48 DEBUG "metric": { 05:25:48 DEBUG }, 05:25:48 DEBUG { 05:25:48 DEBUG 05:25:48 DEBUG 05:25:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:25:48 DEBUG "value": [ 05:25:48 DEBUG "metric": { 05:25:48 DEBUG "pod": "ds-idrepo-2" 05:25:48 DEBUG }, 05:25:48 DEBUG }, 05:25:48 DEBUG 05:25:48 DEBUG 05:25:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:25:48 DEBUG 05:25:48 DEBUG "value": [ 05:25:48 DEBUG "value": [ 05:25:48 DEBUG 05:25:48 DEBUG }, 05:25:48 DEBUG 05:25:48 DEBUG 05:25:48 DEBUG 1713759948, 05:25:48 DEBUG [...] 05:25:48 DEBUG "value": [ 05:25:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:25:48 DEBUG 05:25:48 DEBUG "0" 05:25:48 DEBUG 05:25:48 DEBUG 05:25:48 DEBUG }, 05:25:48 DEBUG 05:25:48 DEBUG ] 05:25:48 DEBUG 05:25:48 DEBUG 05:25:48 DEBUG "value": [ 05:25:48 DEBUG [...] 05:25:48 DEBUG } 05:25:48 DEBUG "instance": "10.142.0.43:9100" 05:25:48 DEBUG 05:25:48 DEBUG 1713759948, 05:25:48 DEBUG 05:25:48 DEBUG ] 05:25:48 DEBUG }, 05:25:48 DEBUG 05:25:48 DEBUG "923.4010000000001" 05:25:48 DEBUG 05:25:48 DEBUG } 05:25:48 DEBUG "value": [ 05:25:48 DEBUG [...] 05:25:48 DEBUG 05:25:48 DEBUG 05:25:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:25:48 DEBUG } 05:25:48 DEBUG 05:25:48 DEBUG ] 05:25:48 DEBUG 1713759948, 05:25:48 DEBUG }, 05:25:48 DEBUG 05:25:48 DEBUG } 05:25:48 DEBUG "0.3795000000000736" 05:25:48 DEBUG "value": [ 05:25:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:25:48 DEBUG ] 05:25:48 DEBUG ] 05:25:48 DEBUG 1713759948, 05:25:48 DEBUG }, 05:25:48 DEBUG } 05:25:48 DEBUG } 05:25:48 DEBUG ] 05:25:48 DEBUG "value": [ 05:25:48 DEBUG 1713759948, 05:25:48 DEBUG } 05:25:48 DEBUG "596229" 05:25:48 DEBUG } 05:25:48 DEBUG "1878.67777727183" 05:25:48 DEBUG } 05:25:48 DEBUG ] 05:25:48 DEBUG ] 05:25:48 DEBUG } 05:25:48 DEBUG } 05:25:48 DEBUG ] 05:25:48 DEBUG ] 05:25:48 DEBUG } 05:25:48 DEBUG } 05:25:48 DEBUG } 05:25:48 DEBUG } 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713759948" 05:25:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713759948" 05:25:48 INFO [http_cmd]: http status code OK 05:25:48 INFO [http_cmd]: http status code OK 05:25:48 INFO [http_cmd]: http status code OK 05:25:48 INFO [http_cmd]: http status code OK 05:25:48 INFO [http_cmd]: http status code OK 05:25:49 INFO [http_cmd]: http status code OK 05:25:49 DEBUG --- status code --- 05:25:49 INFO [http_cmd]: http status code OK 05:25:49 INFO [http_cmd]: http status code OK 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 INFO [http_cmd]: http status code OK 05:25:49 INFO [http_cmd]: http status code OK 05:25:49 INFO [http_cmd]: http status code OK 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG --- status code --- 05:25:49 INFO [http_cmd]: http status code OK 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:49 DEBUG --- http response --- 05:25:49 INFO [http_cmd]: http status code OK 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:49 INFO [http_cmd]: http status code OK 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG --- status code --- 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG { 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG { 05:25:49 DEBUG { 05:25:49 DEBUG { 05:25:49 DEBUG { 05:25:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:25:49 DEBUG { 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:49 DEBUG { 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG { 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG --- http response --- 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "data": { 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "data": { 05:25:49 DEBUG "data": { 05:25:49 DEBUG "data": { 05:25:49 DEBUG "data": { 05:25:49 DEBUG { 05:25:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG { 05:25:49 DEBUG { 05:25:49 DEBUG { 05:25:49 DEBUG "data": { 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG ----- output ----- 05:25:49 DEBUG "data": { 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "data": { 05:25:49 DEBUG { 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "result": [ 05:25:49 DEBUG { 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "result": [ 05:25:49 DEBUG "result": [ 05:25:49 DEBUG "result": [ 05:25:49 DEBUG "result": [ 05:25:49 DEBUG "data": { 05:25:49 DEBUG "data": { 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "data": { 05:25:49 DEBUG "data": { 05:25:49 DEBUG "result": [ 05:25:49 DEBUG { 05:25:49 DEBUG "status": "success", 05:25:49 DEBUG "result": [ 05:25:49 DEBUG { 05:25:49 DEBUG { 05:25:49 DEBUG { 05:25:49 DEBUG { 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "result": [ 05:25:49 DEBUG "data": { 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG { 05:25:49 DEBUG "metric": { 05:25:49 DEBUG "data": { 05:25:49 DEBUG "metric": { 05:25:49 DEBUG { 05:25:49 DEBUG "metric": { 05:25:49 DEBUG "metric": { 05:25:49 DEBUG "metric": { 05:25:49 DEBUG "result": [ 05:25:49 DEBUG "result": [ 05:25:49 DEBUG { 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "result": [ 05:25:49 DEBUG "result": [ 05:25:49 DEBUG "metric": { 05:25:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:25:49 DEBUG "resultType": "vector", 05:25:49 DEBUG "instance": "10.142.0.107:9100" 05:25:49 DEBUG "metric": { 05:25:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:25:49 DEBUG "pod": "ds-idrepo-0" 05:25:49 DEBUG }, 05:25:49 DEBUG { 05:25:49 DEBUG { 05:25:49 DEBUG "metric": { 05:25:49 DEBUG "result": [ 05:25:49 DEBUG { 05:25:49 DEBUG { 05:25:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:25:49 DEBUG }, 05:25:49 DEBUG "result": [ 05:25:49 DEBUG }, 05:25:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:25:49 DEBUG }, 05:25:49 DEBUG "value": [ 05:25:49 DEBUG "instance": "10.142.0.107:9100" 05:25:49 DEBUG "metric": { 05:25:49 DEBUG "metric": { 05:25:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:25:49 DEBUG }, 05:25:49 DEBUG "metric": { 05:25:49 DEBUG "metric": { 05:25:49 DEBUG }, 05:25:49 DEBUG "value": [ 05:25:49 DEBUG { 05:25:49 DEBUG "value": [ 05:25:49 DEBUG }, 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG }, 05:25:49 DEBUG "pod": "ds-idrepo-0" 05:25:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:25:49 DEBUG "value": [ 05:25:49 DEBUG { 05:25:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:25:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG "metric": { 05:25:49 DEBUG 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG "value": [ 05:25:49 DEBUG }, 05:25:49 DEBUG }, 05:25:49 DEBUG "value": [ 05:25:49 DEBUG "metric": { 05:25:49 DEBUG }, 05:25:49 DEBUG }, 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG "instance": "10.142.0.107:9100" 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG "pod": "ds-cts-0" 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG }, 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG }, 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG "instance": "10.142.0.43:9100" 05:25:49 DEBUG 05:25:49 DEBUG "pod": "ds-cts-2" 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG }, 05:25:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:25:49 DEBUG }, 05:25:49 DEBUG 05:25:49 DEBUG "instance": "10.142.0.43:9100" 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG [...] 05:25:49 DEBUG 05:25:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:25:49 DEBUG 05:25:49 DEBUG }, 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:25:49 DEBUG }, 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG }, 05:25:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:25:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:25:49 DEBUG 05:25:49 DEBUG 05:25:49 DEBUG }, 05:25:49 DEBUG 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG }, 05:25:49 DEBUG "value": [ 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG 05:25:49 DEBUG "value": [ 05:25:49 DEBUG }, 05:25:49 DEBUG }, 05:25:49 DEBUG 05:25:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:25:49 DEBUG "value": [ 05:25:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG "0" 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG "value": [ 05:25:49 DEBUG "value": [ 05:25:49 DEBUG 05:25:49 DEBUG }, 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG }, 05:25:49 DEBUG "6637404160" 05:25:49 DEBUG "instance": "10.142.0.43:9100" 05:25:49 DEBUG "0" 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG "0" 05:25:49 DEBUG "pod": "ds-idrepo-2" 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG "pod": "ds-cts-2" 05:25:49 DEBUG "value": [ 05:25:49 DEBUG "15.313260370791179" 05:25:49 DEBUG "value": [ 05:25:49 DEBUG ] 05:25:49 DEBUG }, 05:25:49 DEBUG ] 05:25:49 DEBUG "101533263.26666667" 05:25:49 DEBUG ] 05:25:49 DEBUG }, 05:25:49 DEBUG } 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG "3303.0666666666666" 05:25:49 DEBUG "11.644234498768451" 05:25:49 DEBUG }, 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG ] 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG } 05:25:49 DEBUG "value": [ 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG "101536080.8" 05:25:49 DEBUG ] 05:25:49 DEBUG ] 05:25:49 DEBUG "value": [ 05:25:49 DEBUG "3302.2666666666664" 05:25:49 DEBUG } 05:25:49 DEBUG "3302.1666666666665" 05:25:49 DEBUG ] 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG ] 05:25:49 DEBUG ] 05:25:49 DEBUG "value": [ 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG "0.07133333333332909" 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG ] 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG 1713759948, 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG "0" 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG "0" 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG ] 05:25:49 DEBUG ] 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:25:49 DEBUG } 05:26:06 INFO 05:26:06 INFO [loop_until]: kubectl --namespace=xlou top pods 05:26:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:26:06 INFO [loop_until]: OK (rc = 0) 05:26:06 DEBUG --- stdout --- 05:26:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22579m 8935Mi am-767f6b67b5-qcqz2 23074m 8937Mi am-767f6b67b5-vbltz 24311m 8970Mi ds-cts-0 19823m 12525Mi ds-cts-1 19783m 12566Mi ds-cts-2 19901m 12570Mi ds-idrepo-0 9551m 3994Mi ds-idrepo-1 9146m 3982Mi ds-idrepo-2 9471m 3965Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 7m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11740m 6241Mi 05:26:06 DEBUG --- stderr --- 05:26:06 DEBUG 05:26:13 INFO 05:26:13 INFO [loop_until]: kubectl --namespace=xlou top node 05:26:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:26:13 INFO [loop_until]: OK (rc = 0) 05:26:13 DEBUG --- stdout --- 05:26:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6238Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9885m 33% 8118Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22655m 75% 11639Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3354Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19263m 64% 15786Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 5200Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22973m 76% 11873Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19807m 66% 3265Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25349m 84% 11298Mi 10% gke-xlou-cdm-ds-0f601751-j488 10218m 34% 7491Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19819m 66% 17743Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10016m 33% 8048Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20314m 68% 15270Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11404m 71% 16167Mi 27% 05:26:13 DEBUG --- stderr --- 05:26:13 DEBUG 05:26:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760006" 05:26:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760006" 05:26:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760006" 05:26:46 ERROR [http_cmd]: ERROR 05:26:46 INFO [http_cmd]: http status code OK 05:26:46 ERROR --- status code --- 05:26:46 DEBUG --- status code --- 05:26:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:26:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:46 ERROR --- http response --- 05:26:46 DEBUG --- http response --- 05:26:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:26:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:26:46 INFO [http_cmd]: http status code OK 05:26:46 DEBUG --- status code --- 05:26:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:46 DEBUG --- http response --- 05:26:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:46 DEBUG ----- output ----- 05:26:46 DEBUG { 05:26:46 DEBUG "status": "success", 05:26:46 DEBUG "data": { 05:26:46 DEBUG "resultType": "vector", 05:26:46 DEBUG "result": [ 05:26:46 DEBUG { 05:26:46 DEBUG "metric": { 05:26:46 DEBUG "pod": "ds-idrepo-0" 05:26:46 DEBUG }, 05:26:46 DEBUG "value": [ 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG [...] 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG "pod": "ds-idrepo-2" 05:26:46 DEBUG }, 05:26:46 DEBUG "value": [ 05:26:46 DEBUG 1713760006, 05:26:46 DEBUG "0" 05:26:46 DEBUG ] 05:26:46 DEBUG } 05:26:46 DEBUG ] 05:26:46 DEBUG } 05:26:46 DEBUG } 05:26:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760006" 05:26:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760006" 05:26:46 INFO [http_cmd]: http status code OK 05:26:46 INFO [http_cmd]: http status code OK 05:26:46 DEBUG --- status code --- 05:26:46 DEBUG --- status code --- 05:26:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:46 DEBUG --- http response --- 05:26:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:46 DEBUG --- http response --- 05:26:46 DEBUG ----- output ----- 05:26:46 DEBUG { 05:26:46 DEBUG "status": "success", 05:26:46 DEBUG "data": { 05:26:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:26:46 DEBUG "resultType": "vector", 05:26:46 DEBUG ----- output ----- 05:26:46 DEBUG { 05:26:46 DEBUG "status": "success", 05:26:46 DEBUG "data": { 05:26:46 DEBUG "resultType": "vector", 05:26:46 DEBUG "result": [ 05:26:46 DEBUG { 05:26:46 DEBUG "metric": { 05:26:46 DEBUG "pod": "ds-cts-0" 05:26:46 DEBUG }, 05:26:46 DEBUG "value": [ 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG [...] 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG "pod": "ds-cts-2" 05:26:46 DEBUG }, 05:26:46 DEBUG "value": [ 05:26:46 DEBUG 1713760006, 05:26:46 DEBUG "0.6363999999999881" 05:26:46 DEBUG ] 05:26:46 DEBUG } 05:26:46 DEBUG ] 05:26:46 DEBUG } 05:26:46 DEBUG } 05:26:46 DEBUG "result": [ 05:26:46 DEBUG { 05:26:46 DEBUG "metric": { 05:26:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:26:46 DEBUG }, 05:26:46 DEBUG "value": [ 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG [...] 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG 05:26:46 DEBUG "pod": "ds-cts-0" 05:26:46 DEBUG }, 05:26:46 DEBUG "value": [ 05:26:46 DEBUG 1713760006, 05:26:46 DEBUG "461597384.42608535" 05:26:46 DEBUG ] 05:26:46 DEBUG } 05:26:46 DEBUG ] 05:26:46 DEBUG } 05:26:46 DEBUG } 05:26:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760007" 05:26:47 INFO [http_cmd]: http status code OK 05:26:47 DEBUG --- status code --- 05:26:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:47 DEBUG --- http response --- 05:26:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:47 DEBUG ----- output ----- 05:26:47 DEBUG { 05:26:47 DEBUG "status": "success", 05:26:47 DEBUG "data": { 05:26:47 DEBUG "resultType": "vector", 05:26:47 DEBUG "result": [ 05:26:47 DEBUG { 05:26:47 DEBUG "metric": { 05:26:47 DEBUG "instance": "10.142.0.107:9100" 05:26:47 DEBUG }, 05:26:47 DEBUG "value": [ 05:26:47 DEBUG 05:26:47 DEBUG 05:26:47 DEBUG 05:26:47 DEBUG [...] 05:26:47 DEBUG 05:26:47 DEBUG 05:26:47 DEBUG 05:26:47 DEBUG "instance": "10.142.0.43:9100" 05:26:47 DEBUG }, 05:26:47 DEBUG "value": [ 05:26:47 DEBUG 1713760007, 05:26:47 DEBUG "5779517230080" 05:26:47 DEBUG ] 05:26:47 DEBUG } 05:26:47 DEBUG ] 05:26:47 DEBUG } 05:26:47 DEBUG } 05:26:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760008" 05:26:48 INFO [http_cmd]: http status code OK 05:26:48 DEBUG --- status code --- 05:26:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:48 DEBUG --- http response --- 05:26:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:48 DEBUG ----- output ----- 05:26:48 DEBUG { 05:26:48 DEBUG "status": "success", 05:26:48 DEBUG "data": { 05:26:48 DEBUG "resultType": "vector", 05:26:48 DEBUG "result": [ 05:26:48 DEBUG { 05:26:48 DEBUG "metric": { 05:26:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:26:48 DEBUG }, 05:26:48 DEBUG "value": [ 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG [...] 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:26:48 DEBUG }, 05:26:48 DEBUG "value": [ 05:26:48 DEBUG 1713760008, 05:26:48 DEBUG "4317.833333333333" 05:26:48 DEBUG ] 05:26:48 DEBUG } 05:26:48 DEBUG ] 05:26:48 DEBUG } 05:26:48 DEBUG } 05:26:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760008" 05:26:48 INFO [http_cmd]: http status code OK 05:26:48 DEBUG --- status code --- 05:26:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:48 DEBUG --- http response --- 05:26:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:26:48 DEBUG ----- output ----- 05:26:48 DEBUG { 05:26:48 DEBUG "status": "success", 05:26:48 DEBUG "data": { 05:26:48 DEBUG "resultType": "vector", 05:26:48 DEBUG "result": [ 05:26:48 DEBUG { 05:26:48 DEBUG "metric": { 05:26:48 DEBUG "pod": "ds-cts-0" 05:26:48 DEBUG }, 05:26:48 DEBUG "value": [ 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG [...] 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG "pod": "ds-cts-2" 05:26:48 DEBUG }, 05:26:48 DEBUG "value": [ 05:26:48 DEBUG 1713760008, 05:26:48 DEBUG "23.166666666666668" 05:26:48 DEBUG ] 05:26:48 DEBUG } 05:26:48 DEBUG ] 05:26:48 DEBUG } 05:26:48 DEBUG } 05:26:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760008" 05:26:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760008" 05:26:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760008" 05:26:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760008" 05:26:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760008" 05:26:48 INFO [http_cmd]: http status code OK 05:26:48 DEBUG --- status code --- 05:26:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:48 DEBUG --- http response --- 05:26:48 INFO [http_cmd]: http status code OK 05:26:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:48 DEBUG --- status code --- 05:26:48 DEBUG ----- output ----- 05:26:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:48 DEBUG { 05:26:48 DEBUG --- http response --- 05:26:48 DEBUG "status": "success", 05:26:48 INFO [http_cmd]: http status code OK 05:26:48 DEBUG "data": { 05:26:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:48 DEBUG --- status code --- 05:26:48 DEBUG "resultType": "vector", 05:26:48 DEBUG ----- output ----- 05:26:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:48 DEBUG "result": [ 05:26:48 INFO [http_cmd]: http status code OK 05:26:48 DEBUG { 05:26:48 INFO [http_cmd]: http status code OK 05:26:48 DEBUG --- http response --- 05:26:48 DEBUG { 05:26:48 DEBUG --- status code --- 05:26:48 DEBUG "status": "success", 05:26:48 DEBUG --- status code --- 05:26:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:48 DEBUG "metric": { 05:26:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:48 DEBUG "data": { 05:26:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:48 DEBUG ----- output ----- 05:26:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:26:48 DEBUG --- http response --- 05:26:48 DEBUG "resultType": "vector", 05:26:48 DEBUG --- http response --- 05:26:48 DEBUG { 05:26:48 DEBUG }, 05:26:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:48 DEBUG "result": [ 05:26:48 DEBUG "status": "success", 05:26:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:26:48 DEBUG "value": [ 05:26:48 DEBUG ----- output ----- 05:26:48 DEBUG { 05:26:48 DEBUG "metric": { 05:26:48 DEBUG "data": { 05:26:48 DEBUG 05:26:48 DEBUG { 05:26:48 DEBUG ----- output ----- 05:26:48 DEBUG "pod": "ds-idrepo-0" 05:26:48 DEBUG "resultType": "vector", 05:26:48 DEBUG 05:26:48 DEBUG "status": "success", 05:26:48 DEBUG { 05:26:48 DEBUG }, 05:26:48 DEBUG "result": [ 05:26:48 DEBUG 05:26:48 DEBUG "data": { 05:26:48 DEBUG "status": "success", 05:26:48 DEBUG "value": [ 05:26:48 DEBUG { 05:26:48 DEBUG [...] 05:26:48 DEBUG "resultType": "vector", 05:26:48 DEBUG "data": { 05:26:48 DEBUG 05:26:48 DEBUG "metric": { 05:26:48 DEBUG 05:26:48 DEBUG "result": [ 05:26:48 DEBUG "resultType": "vector", 05:26:48 DEBUG 05:26:48 DEBUG "instance": "10.142.0.107:9100" 05:26:48 DEBUG 05:26:48 DEBUG { 05:26:48 DEBUG "result": [ 05:26:48 DEBUG 05:26:48 DEBUG }, 05:26:48 DEBUG 05:26:48 DEBUG "metric": { 05:26:48 DEBUG { 05:26:48 DEBUG [...] 05:26:48 DEBUG "value": [ 05:26:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:26:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:26:48 DEBUG "metric": { 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG }, 05:26:48 DEBUG }, 05:26:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG "value": [ 05:26:48 DEBUG "value": [ 05:26:48 DEBUG }, 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG 1713760008, 05:26:48 DEBUG 05:26:48 DEBUG "value": [ 05:26:48 DEBUG "pod": "ds-idrepo-2" 05:26:48 DEBUG [...] 05:26:48 DEBUG "933.636" 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG }, 05:26:48 DEBUG 05:26:48 DEBUG ] 05:26:48 DEBUG [...] 05:26:48 DEBUG "value": [ 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG } 05:26:48 DEBUG 05:26:48 DEBUG 1713760008, 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG [...] 05:26:48 DEBUG ] 05:26:48 DEBUG } 05:26:48 DEBUG } 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG "instance": "10.142.0.43:9100" 05:26:48 DEBUG 05:26:48 DEBUG 05:26:48 DEBUG }, 05:26:48 DEBUG 05:26:48 DEBUG "0" 05:26:48 DEBUG 05:26:48 DEBUG "value": [ 05:26:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:26:48 DEBUG ] 05:26:48 DEBUG "pod": "ds-cts-0" 05:26:48 DEBUG 1713760008, 05:26:48 DEBUG }, 05:26:48 DEBUG } 05:26:48 DEBUG }, 05:26:48 DEBUG "0.3760333333335154" 05:26:48 DEBUG "value": [ 05:26:48 DEBUG ] 05:26:48 DEBUG "value": [ 05:26:48 DEBUG 1713760008, 05:26:48 DEBUG 1713760008, 05:26:48 DEBUG } 05:26:48 DEBUG ] 05:26:48 DEBUG "82744959.88750218" 05:26:48 DEBUG "598142" 05:26:48 DEBUG } 05:26:48 DEBUG } 05:26:48 DEBUG ] 05:26:48 DEBUG ] 05:26:48 DEBUG ] 05:26:48 DEBUG } 05:26:48 DEBUG ] 05:26:48 DEBUG } 05:26:48 DEBUG } 05:26:48 DEBUG } 05:26:48 DEBUG } 05:26:48 DEBUG ] 05:26:48 DEBUG } 05:26:48 DEBUG } 05:26:48 DEBUG } 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760009" 05:26:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760009" 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG --- status code --- 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 DEBUG --- status code --- 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG --- http response --- 05:26:49 INFO [http_cmd]: http status code OK 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG { 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:49 DEBUG --- status code --- 05:26:49 DEBUG { 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG { 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG { 05:26:49 DEBUG { 05:26:49 DEBUG { 05:26:49 DEBUG { 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG "data": { 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG { 05:26:49 DEBUG { 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG --- http response --- 05:26:49 DEBUG "data": { 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG { 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG "data": { 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG "result": [ 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG "data": { 05:26:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG { 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG "data": { 05:26:49 DEBUG "data": { 05:26:49 DEBUG "data": { 05:26:49 DEBUG "data": { 05:26:49 DEBUG { 05:26:49 DEBUG "result": [ 05:26:49 DEBUG { 05:26:49 DEBUG "data": { 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG "result": [ 05:26:49 DEBUG ----- output ----- 05:26:49 DEBUG "result": [ 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG "data": { 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG { 05:26:49 DEBUG "metric": { 05:26:49 DEBUG { 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG { 05:26:49 DEBUG { 05:26:49 DEBUG { 05:26:49 DEBUG "data": { 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG "result": [ 05:26:49 DEBUG "result": [ 05:26:49 DEBUG "result": [ 05:26:49 DEBUG "result": [ 05:26:49 DEBUG "data": { 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "result": [ 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG "status": "success", 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG "result": [ 05:26:49 DEBUG { 05:26:49 DEBUG { 05:26:49 DEBUG "metric": { 05:26:49 DEBUG { 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:26:49 DEBUG }, 05:26:49 DEBUG "pod": "ds-idrepo-0" 05:26:49 DEBUG { 05:26:49 DEBUG "data": { 05:26:49 DEBUG "data": { 05:26:49 DEBUG "pod": "ds-cts-0" 05:26:49 DEBUG "result": [ 05:26:49 DEBUG { 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:26:49 DEBUG { 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "result": [ 05:26:49 DEBUG }, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG }, 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG "resultType": "vector", 05:26:49 DEBUG }, 05:26:49 DEBUG { 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:26:49 DEBUG }, 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "instance": "10.142.0.107:9100" 05:26:49 DEBUG { 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 05:26:49 DEBUG "value": [ 05:26:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:26:49 DEBUG "result": [ 05:26:49 DEBUG "result": [ 05:26:49 DEBUG "value": [ 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:26:49 DEBUG }, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG "pod": "ds-idrepo-0" 05:26:49 DEBUG }, 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "instance": "10.142.0.107:9100" 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG }, 05:26:49 DEBUG { 05:26:49 DEBUG { 05:26:49 DEBUG 05:26:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:26:49 DEBUG }, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 05:26:49 DEBUG }, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG }, 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "value": [ 05:26:49 DEBUG "metric": { 05:26:49 DEBUG "metric": { 05:26:49 DEBUG 05:26:49 DEBUG }, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:26:49 DEBUG "instance": "10.142.0.107:9100" 05:26:49 DEBUG 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG [...] 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG }, 05:26:49 DEBUG }, 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "value": [ 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG [...] 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "pod": "ds-idrepo-2" 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG [...] 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "pod": "ds-cts-0" 05:26:49 DEBUG }, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "pod": "ds-cts-2" 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "instance": "10.142.0.43:9100" 05:26:49 DEBUG }, 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "instance": "10.142.0.43:9100" 05:26:49 DEBUG }, 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG [...] 05:26:49 DEBUG 05:26:49 DEBUG }, 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG 05:26:49 DEBUG "instance": "10.142.0.43:9100" 05:26:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:26:49 DEBUG }, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG "0" 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG }, 05:26:49 DEBUG "0.07500000000000757" 05:26:49 DEBUG "pod": "ds-cts-2" 05:26:49 DEBUG 05:26:49 DEBUG }, 05:26:49 DEBUG }, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG ] 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG 05:26:49 DEBUG 05:26:49 DEBUG "value": [ 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG "value": [ 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG "20.14879328528598" 05:26:49 DEBUG } 05:26:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:26:49 DEBUG 05:26:49 DEBUG "0" 05:26:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:26:49 DEBUG }, 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG } 05:26:49 DEBUG }, 05:26:49 DEBUG 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG "100890370.80000001" 05:26:49 DEBUG ] 05:26:49 DEBUG ] 05:26:49 DEBUG }, 05:26:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:26:49 DEBUG ] 05:26:49 DEBUG "value": [ 05:26:49 DEBUG "11.583930388463259" 05:26:49 DEBUG } 05:26:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:26:49 DEBUG }, 05:26:49 DEBUG "pod": "ds-cts-0" 05:26:49 DEBUG "100891763.19999999" 05:26:49 DEBUG ] 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG "value": [ 05:26:49 DEBUG }, 05:26:49 DEBUG } 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG ] 05:26:49 DEBUG "value": [ 05:26:49 DEBUG }, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG } 05:26:49 DEBUG "0" 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG ] 05:26:49 DEBUG "6642339840" 05:26:49 DEBUG } 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG "value": [ 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG ] 05:26:49 DEBUG ] 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG "3317.366666666667" 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG "0" 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG "3321.2999999999997" 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG "3318.8" 05:26:49 DEBUG 1713760009, 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG "74448849.92089997" 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG ] 05:26:49 DEBUG ] 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG } 05:26:49 DEBUG } 05:27:06 INFO 05:27:06 INFO [loop_until]: kubectl --namespace=xlou top pods 05:27:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:27:06 INFO [loop_until]: OK (rc = 0) 05:27:06 DEBUG --- stdout --- 05:27:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22604m 8928Mi am-767f6b67b5-qcqz2 22657m 8937Mi am-767f6b67b5-vbltz 25021m 8908Mi ds-cts-0 19798m 12523Mi ds-cts-1 19901m 12560Mi ds-cts-2 19770m 12571Mi ds-idrepo-0 9839m 3994Mi ds-idrepo-1 9477m 3982Mi ds-idrepo-2 9918m 3965Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11851m 6243Mi 05:27:06 DEBUG --- stderr --- 05:27:06 DEBUG 05:27:13 INFO 05:27:13 INFO [loop_until]: kubectl --namespace=xlou top node 05:27:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:27:13 INFO [loop_until]: OK (rc = 0) 05:27:13 DEBUG --- stdout --- 05:27:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6233Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10016m 33% 8113Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22695m 76% 11690Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3346Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 20199m 67% 15800Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5218Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22943m 76% 11868Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 20047m 67% 3265Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25338m 84% 11232Mi 10% gke-xlou-cdm-ds-0f601751-j488 10327m 34% 7469Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19946m 66% 17742Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10079m 33% 8053Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20071m 67% 15305Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11939m 75% 16172Mi 27% 05:27:13 DEBUG --- stderr --- 05:27:13 DEBUG 05:27:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760066" 05:27:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760066" 05:27:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760066" 05:27:46 INFO [http_cmd]: http status code OK 05:27:46 DEBUG --- status code --- 05:27:46 ERROR [http_cmd]: ERROR 05:27:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:46 ERROR --- status code --- 05:27:46 DEBUG --- http response --- 05:27:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:27:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:46 ERROR --- http response --- 05:27:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:27:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:27:46 INFO [http_cmd]: http status code OK 05:27:46 DEBUG --- status code --- 05:27:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:46 DEBUG --- http response --- 05:27:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:46 DEBUG ----- output ----- 05:27:46 DEBUG { 05:27:46 DEBUG "status": "success", 05:27:46 DEBUG "data": { 05:27:46 DEBUG "resultType": "vector", 05:27:46 DEBUG "result": [ 05:27:46 DEBUG { 05:27:46 DEBUG "metric": { 05:27:46 DEBUG "pod": "ds-idrepo-0" 05:27:46 DEBUG }, 05:27:46 DEBUG "value": [ 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG [...] 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG "pod": "ds-idrepo-2" 05:27:46 DEBUG }, 05:27:46 DEBUG "value": [ 05:27:46 DEBUG 1713760066, 05:27:46 DEBUG "0" 05:27:46 DEBUG ] 05:27:46 DEBUG } 05:27:46 DEBUG ] 05:27:46 DEBUG } 05:27:46 DEBUG } 05:27:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760066" 05:27:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760066" 05:27:46 INFO [http_cmd]: http status code OK 05:27:46 DEBUG --- status code --- 05:27:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:46 DEBUG --- http response --- 05:27:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:27:46 DEBUG ----- output ----- 05:27:46 DEBUG { 05:27:46 DEBUG "status": "success", 05:27:46 DEBUG "data": { 05:27:46 DEBUG "resultType": "vector", 05:27:46 DEBUG "result": [ 05:27:46 DEBUG { 05:27:46 DEBUG "metric": { 05:27:46 DEBUG "pod": "ds-cts-0" 05:27:46 DEBUG }, 05:27:46 DEBUG "value": [ 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG [...] 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG "pod": "ds-cts-2" 05:27:46 DEBUG }, 05:27:46 DEBUG "value": [ 05:27:46 DEBUG 1713760066, 05:27:46 DEBUG "0.6615333333333335" 05:27:46 DEBUG ] 05:27:46 DEBUG } 05:27:46 DEBUG ] 05:27:46 DEBUG } 05:27:46 DEBUG } 05:27:46 INFO [http_cmd]: http status code OK 05:27:46 DEBUG --- status code --- 05:27:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:46 DEBUG --- http response --- 05:27:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:27:46 DEBUG ----- output ----- 05:27:46 DEBUG { 05:27:46 DEBUG "status": "success", 05:27:46 DEBUG "data": { 05:27:46 DEBUG "resultType": "vector", 05:27:46 DEBUG "result": [ 05:27:46 DEBUG { 05:27:46 DEBUG "metric": { 05:27:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:27:46 DEBUG }, 05:27:46 DEBUG "value": [ 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG [...] 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG 05:27:46 DEBUG "pod": "ds-cts-2" 05:27:46 DEBUG }, 05:27:46 DEBUG "value": [ 05:27:46 DEBUG 1713760066, 05:27:46 DEBUG "363555941.9209754" 05:27:46 DEBUG ] 05:27:46 DEBUG } 05:27:46 DEBUG ] 05:27:46 DEBUG } 05:27:46 DEBUG } 05:27:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760067" 05:27:47 INFO [http_cmd]: http status code OK 05:27:47 DEBUG --- status code --- 05:27:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:47 DEBUG --- http response --- 05:27:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:47 DEBUG ----- output ----- 05:27:47 DEBUG { 05:27:47 DEBUG "status": "success", 05:27:47 DEBUG "data": { 05:27:47 DEBUG "resultType": "vector", 05:27:47 DEBUG "result": [ 05:27:47 DEBUG { 05:27:47 DEBUG "metric": { 05:27:47 DEBUG "instance": "10.142.0.107:9100" 05:27:47 DEBUG }, 05:27:47 DEBUG "value": [ 05:27:47 DEBUG 05:27:47 DEBUG 05:27:47 DEBUG 05:27:47 DEBUG [...] 05:27:47 DEBUG 05:27:47 DEBUG 05:27:47 DEBUG 05:27:47 DEBUG "instance": "10.142.0.43:9100" 05:27:47 DEBUG }, 05:27:47 DEBUG "value": [ 05:27:47 DEBUG 1713760067, 05:27:47 DEBUG "5784636337152" 05:27:47 DEBUG ] 05:27:47 DEBUG } 05:27:47 DEBUG ] 05:27:47 DEBUG } 05:27:47 DEBUG } 05:27:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760068" 05:27:48 INFO [http_cmd]: http status code OK 05:27:48 DEBUG --- status code --- 05:27:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:48 DEBUG --- http response --- 05:27:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:48 DEBUG ----- output ----- 05:27:48 DEBUG { 05:27:48 DEBUG "status": "success", 05:27:48 DEBUG "data": { 05:27:48 DEBUG "resultType": "vector", 05:27:48 DEBUG "result": [ 05:27:48 DEBUG { 05:27:48 DEBUG "metric": { 05:27:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:27:48 DEBUG }, 05:27:48 DEBUG "value": [ 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG [...] 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:27:48 DEBUG }, 05:27:48 DEBUG "value": [ 05:27:48 DEBUG 1713760068, 05:27:48 DEBUG "4314.233333333334" 05:27:48 DEBUG ] 05:27:48 DEBUG } 05:27:48 DEBUG ] 05:27:48 DEBUG } 05:27:48 DEBUG } 05:27:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760068" 05:27:48 INFO [http_cmd]: http status code OK 05:27:48 DEBUG --- status code --- 05:27:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:48 DEBUG --- http response --- 05:27:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:27:48 DEBUG ----- output ----- 05:27:48 DEBUG { 05:27:48 DEBUG "status": "success", 05:27:48 DEBUG "data": { 05:27:48 DEBUG "resultType": "vector", 05:27:48 DEBUG "result": [ 05:27:48 DEBUG { 05:27:48 DEBUG "metric": { 05:27:48 DEBUG "pod": "ds-cts-0" 05:27:48 DEBUG }, 05:27:48 DEBUG "value": [ 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG [...] 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG "pod": "ds-cts-2" 05:27:48 DEBUG }, 05:27:48 DEBUG "value": [ 05:27:48 DEBUG 1713760068, 05:27:48 DEBUG "22.366666666666667" 05:27:48 DEBUG ] 05:27:48 DEBUG } 05:27:48 DEBUG ] 05:27:48 DEBUG } 05:27:48 DEBUG } 05:27:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760068" 05:27:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760068" 05:27:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760068" 05:27:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760068" 05:27:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760068" 05:27:48 INFO [http_cmd]: http status code OK 05:27:48 DEBUG --- status code --- 05:27:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:48 DEBUG --- http response --- 05:27:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:48 DEBUG ----- output ----- 05:27:48 DEBUG { 05:27:48 DEBUG "status": "success", 05:27:48 DEBUG "data": { 05:27:48 DEBUG "resultType": "vector", 05:27:48 DEBUG "result": [ 05:27:48 DEBUG { 05:27:48 DEBUG "metric": { 05:27:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:27:48 DEBUG }, 05:27:48 DEBUG "value": [ 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG [...] 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:27:48 DEBUG }, 05:27:48 DEBUG "value": [ 05:27:48 DEBUG 1713760068, 05:27:48 DEBUG "945.315" 05:27:48 DEBUG ] 05:27:48 DEBUG } 05:27:48 DEBUG ] 05:27:48 INFO [http_cmd]: http status code OK 05:27:48 DEBUG } 05:27:48 DEBUG --- status code --- 05:27:48 DEBUG } 05:27:48 INFO [http_cmd]: http status code OK 05:27:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:48 INFO [http_cmd]: http status code OK 05:27:48 DEBUG --- status code --- 05:27:48 DEBUG --- http response --- 05:27:48 DEBUG --- status code --- 05:27:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:48 INFO [http_cmd]: http status code OK 05:27:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:27:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:48 DEBUG --- http response --- 05:27:48 DEBUG --- status code --- 05:27:48 DEBUG ----- output ----- 05:27:48 DEBUG --- http response --- 05:27:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:48 DEBUG { 05:27:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:48 DEBUG ----- output ----- 05:27:48 DEBUG --- http response --- 05:27:48 DEBUG "status": "success", 05:27:48 DEBUG ----- output ----- 05:27:48 DEBUG { 05:27:48 DEBUG "data": { 05:27:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:48 DEBUG { 05:27:48 DEBUG "status": "success", 05:27:48 DEBUG "resultType": "vector", 05:27:48 DEBUG ----- output ----- 05:27:48 DEBUG "status": "success", 05:27:48 DEBUG "data": { 05:27:48 DEBUG "result": [ 05:27:48 DEBUG { 05:27:48 DEBUG "data": { 05:27:48 DEBUG "resultType": "vector", 05:27:48 DEBUG { 05:27:48 DEBUG "status": "success", 05:27:48 DEBUG "resultType": "vector", 05:27:48 DEBUG "result": [ 05:27:48 DEBUG "metric": { 05:27:48 DEBUG "data": { 05:27:48 DEBUG "resultType": "vector", 05:27:48 DEBUG "result": [ 05:27:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:27:48 DEBUG { 05:27:48 DEBUG "result": [ 05:27:48 DEBUG { 05:27:48 DEBUG }, 05:27:48 DEBUG "metric": { 05:27:48 DEBUG { 05:27:48 DEBUG "metric": { 05:27:48 DEBUG "value": [ 05:27:48 DEBUG "pod": "ds-idrepo-0" 05:27:48 DEBUG "metric": { 05:27:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:27:48 DEBUG 05:27:48 DEBUG }, 05:27:48 DEBUG "instance": "10.142.0.107:9100" 05:27:48 DEBUG }, 05:27:48 DEBUG 05:27:48 DEBUG "value": [ 05:27:48 DEBUG "value": [ 05:27:48 DEBUG 05:27:48 DEBUG [...] 05:27:48 DEBUG }, 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG "value": [ 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG [...] 05:27:48 DEBUG [...] 05:27:48 DEBUG "pod": "ds-cts-2" 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG }, 05:27:48 DEBUG "value": [ 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG 05:27:48 DEBUG "pod": "ds-idrepo-2" 05:27:48 DEBUG }, 05:27:48 DEBUG "value": [ 05:27:48 DEBUG 1713760068, 05:27:48 DEBUG "0" 05:27:48 DEBUG ] 05:27:48 DEBUG 05:27:48 DEBUG } 05:27:48 DEBUG [...] 05:27:48 DEBUG 1713760068, 05:27:48 DEBUG "instance": "10.142.0.43:9100" 05:27:48 DEBUG ] 05:27:48 DEBUG 05:27:48 DEBUG "43145105.40188961" 05:27:48 DEBUG }, 05:27:48 DEBUG } 05:27:48 DEBUG 05:27:48 DEBUG ] 05:27:48 DEBUG "value": [ 05:27:48 DEBUG } 05:27:48 DEBUG 05:27:48 DEBUG } 05:27:48 DEBUG 1713760068, 05:27:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:27:48 DEBUG ] 05:27:48 DEBUG "0.38643333333353097" 05:27:48 DEBUG }, 05:27:48 DEBUG } 05:27:48 DEBUG ] 05:27:48 DEBUG "value": [ 05:27:48 DEBUG } 05:27:48 DEBUG } 05:27:48 DEBUG 1713760068, 05:27:48 DEBUG "595511" 05:27:48 DEBUG ] 05:27:48 DEBUG ] 05:27:48 DEBUG } 05:27:48 DEBUG } 05:27:48 DEBUG } 05:27:48 DEBUG ] 05:27:48 DEBUG } 05:27:48 DEBUG } 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760069" 05:27:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760069" 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 DEBUG --- status code --- 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:49 INFO [http_cmd]: http status code OK 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:27:49 DEBUG --- status code --- 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:49 DEBUG { 05:27:49 DEBUG { 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG { 05:27:49 DEBUG --- http response --- 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG { 05:27:49 DEBUG { 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:49 DEBUG { 05:27:49 DEBUG { 05:27:49 DEBUG { 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG { 05:27:49 DEBUG "data": { 05:27:49 DEBUG "data": { 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:49 DEBUG { 05:27:49 DEBUG { 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG { 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "result": [ 05:27:49 DEBUG { 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "data": { 05:27:49 DEBUG "data": { 05:27:49 DEBUG { 05:27:49 DEBUG "data": { 05:27:49 DEBUG "data": { 05:27:49 DEBUG "data": { 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "data": { 05:27:49 DEBUG "result": [ 05:27:49 DEBUG { 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "data": { 05:27:49 DEBUG "data": { 05:27:49 DEBUG "data": { 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "data": { 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "metric": { 05:27:49 DEBUG ----- output ----- 05:27:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "result": [ 05:27:49 DEBUG "result": [ 05:27:49 DEBUG "data": { 05:27:49 DEBUG "result": [ 05:27:49 DEBUG "result": [ 05:27:49 DEBUG "result": [ 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "result": [ 05:27:49 DEBUG "pod": "ds-idrepo-0" 05:27:49 DEBUG { 05:27:49 DEBUG }, 05:27:49 DEBUG "result": [ 05:27:49 DEBUG "result": [ 05:27:49 DEBUG "result": [ 05:27:49 DEBUG { 05:27:49 DEBUG { 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG { 05:27:49 DEBUG { 05:27:49 DEBUG { 05:27:49 DEBUG "result": [ 05:27:49 DEBUG { 05:27:49 DEBUG }, 05:27:49 DEBUG "status": "success", 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 05:27:49 DEBUG { 05:27:49 DEBUG { 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "instance": "10.142.0.107:9100" 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "pod": "ds-idrepo-0" 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "value": [ 05:27:49 DEBUG "data": { 05:27:49 DEBUG { 05:27:49 DEBUG 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:27:49 DEBUG }, 05:27:49 DEBUG "result": [ 05:27:49 DEBUG "instance": "10.142.0.107:9100" 05:27:49 DEBUG }, 05:27:49 DEBUG { 05:27:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:27:49 DEBUG "instance": "10.142.0.107:9100" 05:27:49 DEBUG 05:27:49 DEBUG "resultType": "vector", 05:27:49 DEBUG "metric": { 05:27:49 DEBUG 05:27:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:27:49 DEBUG }, 05:27:49 DEBUG }, 05:27:49 DEBUG "value": [ 05:27:49 DEBUG { 05:27:49 DEBUG }, 05:27:49 DEBUG "value": [ 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:27:49 DEBUG }, 05:27:49 DEBUG 05:27:49 DEBUG "result": [ 05:27:49 DEBUG "pod": "ds-cts-0" 05:27:49 DEBUG [...] 05:27:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:27:49 DEBUG "value": [ 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 05:27:49 DEBUG }, 05:27:49 DEBUG }, 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 05:27:49 DEBUG { 05:27:49 DEBUG }, 05:27:49 DEBUG 05:27:49 DEBUG }, 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG "value": [ 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 05:27:49 DEBUG [...] 05:27:49 DEBUG "metric": { 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG }, 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG [...] 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 05:27:49 DEBUG [...] 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:27:49 DEBUG 05:27:49 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:27:49 DEBUG 05:27:49 DEBUG [...] 05:27:49 DEBUG [...] 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG [...] 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG [...] 05:27:49 DEBUG "pod": "ds-idrepo-2" 05:27:49 DEBUG }, 05:27:49 DEBUG 05:27:49 DEBUG }, 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG [...] 05:27:49 DEBUG [...] 05:27:49 DEBUG 05:27:49 DEBUG }, 05:27:49 DEBUG "value": [ 05:27:49 DEBUG [...] 05:27:49 DEBUG "value": [ 05:27:49 DEBUG [...] 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG "instance": "10.142.0.43:9100" 05:27:49 DEBUG [...] 05:27:49 DEBUG 05:27:49 DEBUG "pod": "ds-cts-2" 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG "20.210394618465457" 05:27:49 DEBUG 05:27:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:27:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:27:49 DEBUG }, 05:27:49 DEBUG 05:27:49 DEBUG "instance": "10.142.0.43:9100" 05:27:49 DEBUG }, 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG "instance": "10.142.0.43:9100" 05:27:49 DEBUG "0" 05:27:49 DEBUG 05:27:49 DEBUG 05:27:49 DEBUG ] 05:27:49 DEBUG 05:27:49 DEBUG }, 05:27:49 DEBUG }, 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 05:27:49 DEBUG }, 05:27:49 DEBUG "value": [ 05:27:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:27:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:27:49 DEBUG }, 05:27:49 DEBUG ] 05:27:49 DEBUG [...] 05:27:49 DEBUG "pod": "ds-cts-2" 05:27:49 DEBUG } 05:27:49 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:27:49 DEBUG "value": [ 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG }, 05:27:49 DEBUG }, 05:27:49 DEBUG "value": [ 05:27:49 DEBUG } 05:27:49 DEBUG 05:27:49 DEBUG }, 05:27:49 DEBUG ] 05:27:49 DEBUG }, 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG "102705560.93333334" 05:27:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG "0" 05:27:49 DEBUG "value": [ 05:27:49 DEBUG "value": [ 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG "0.07399999999999333" 05:27:49 DEBUG 05:27:49 DEBUG "value": [ 05:27:49 DEBUG } 05:27:49 DEBUG "value": [ 05:27:49 DEBUG "0" 05:27:49 DEBUG "11.666905715321139" 05:27:49 DEBUG ] 05:27:49 DEBUG }, 05:27:49 DEBUG "102708718.73333332" 05:27:49 DEBUG ] 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG ] 05:27:49 DEBUG ] 05:27:49 DEBUG 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG } 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG ] 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG "value": [ 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG "3242.4" 05:27:49 DEBUG "6656040960" 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:27:49 DEBUG "0" 05:27:49 DEBUG "0" 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG ] 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG } 05:27:49 DEBUG ] 05:27:49 DEBUG ] 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG ] 05:27:49 DEBUG }, 05:27:49 DEBUG ] 05:27:49 DEBUG ] 05:27:49 DEBUG ] 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG "3235.8333333333335" 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG "value": [ 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG 1713760069, 05:27:49 DEBUG ] 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG "3238.2" 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG } 05:27:49 DEBUG ] 05:27:49 DEBUG } 05:27:49 DEBUG } 05:28:06 INFO 05:28:06 INFO [loop_until]: kubectl --namespace=xlou top pods 05:28:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:28:06 INFO [loop_until]: OK (rc = 0) 05:28:06 DEBUG --- stdout --- 05:28:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 23441m 8899Mi am-767f6b67b5-qcqz2 23199m 8952Mi am-767f6b67b5-vbltz 24427m 8941Mi ds-cts-0 20384m 12525Mi ds-cts-1 19857m 12559Mi ds-cts-2 20115m 12571Mi ds-idrepo-0 9776m 3994Mi ds-idrepo-1 9738m 3982Mi ds-idrepo-2 9516m 3965Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 3m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11539m 6250Mi 05:28:06 DEBUG --- stderr --- 05:28:06 DEBUG 05:28:13 INFO 05:28:13 INFO [loop_until]: kubectl --namespace=xlou top node 05:28:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:28:14 INFO [loop_until]: OK (rc = 0) 05:28:14 DEBUG --- stdout --- 05:28:14 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 6239Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10020m 33% 8109Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22868m 76% 11651Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3349Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20280m 67% 15795Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 23309m 78% 11873Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 20142m 67% 3281Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25257m 84% 11258Mi 10% gke-xlou-cdm-ds-0f601751-j488 10303m 34% 7485Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19922m 66% 17742Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10123m 33% 8039Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20240m 67% 15313Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11953m 75% 16175Mi 27% 05:28:14 DEBUG --- stderr --- 05:28:14 DEBUG 05:28:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760126" 05:28:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760126" 05:28:46 INFO [http_cmd]: http status code OK 05:28:46 DEBUG --- status code --- 05:28:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:46 DEBUG --- http response --- 05:28:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:46 DEBUG ----- output ----- 05:28:46 DEBUG { 05:28:46 DEBUG "status": "success", 05:28:46 DEBUG "data": { 05:28:46 DEBUG "resultType": "vector", 05:28:46 DEBUG "result": [ 05:28:46 DEBUG { 05:28:46 DEBUG "metric": { 05:28:46 DEBUG "pod": "ds-idrepo-0" 05:28:46 DEBUG }, 05:28:46 DEBUG "value": [ 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG [...] 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG "pod": "ds-idrepo-2" 05:28:46 DEBUG }, 05:28:46 DEBUG "value": [ 05:28:46 DEBUG 1713760126, 05:28:46 DEBUG "0" 05:28:46 DEBUG ] 05:28:46 DEBUG } 05:28:46 DEBUG ] 05:28:46 DEBUG } 05:28:46 DEBUG } 05:28:46 INFO [http_cmd]: http status code OK 05:28:46 DEBUG --- status code --- 05:28:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:46 DEBUG --- http response --- 05:28:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:28:46 DEBUG ----- output ----- 05:28:46 DEBUG { 05:28:46 DEBUG "status": "success", 05:28:46 DEBUG "data": { 05:28:46 DEBUG "resultType": "vector", 05:28:46 DEBUG "result": [ 05:28:46 DEBUG { 05:28:46 DEBUG "metric": { 05:28:46 DEBUG "pod": "ds-cts-0" 05:28:46 DEBUG }, 05:28:46 DEBUG "value": [ 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG [...] 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG "pod": "ds-cts-2" 05:28:46 DEBUG }, 05:28:46 DEBUG "value": [ 05:28:46 DEBUG 1713760126, 05:28:46 DEBUG "0.01663333333333412" 05:28:46 DEBUG ] 05:28:46 DEBUG } 05:28:46 DEBUG ] 05:28:46 DEBUG } 05:28:46 DEBUG } 05:28:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760126" 05:28:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760126" 05:28:46 ERROR [http_cmd]: ERROR 05:28:46 ERROR --- status code --- 05:28:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:28:46 ERROR --- http response --- 05:28:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:28:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:28:46 INFO [http_cmd]: http status code OK 05:28:46 DEBUG --- status code --- 05:28:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:46 DEBUG --- http response --- 05:28:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760126" 05:28:46 INFO [http_cmd]: http status code OK 05:28:46 DEBUG --- status code --- 05:28:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:46 DEBUG --- http response --- 05:28:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:28:46 DEBUG ----- output ----- 05:28:46 DEBUG { 05:28:46 DEBUG "status": "success", 05:28:46 DEBUG "data": { 05:28:46 DEBUG "resultType": "vector", 05:28:46 DEBUG "result": [ 05:28:46 DEBUG { 05:28:46 DEBUG "metric": { 05:28:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:28:46 DEBUG }, 05:28:46 DEBUG "value": [ 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG [...] 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG 05:28:46 DEBUG "pod": "ds-cts-2" 05:28:46 DEBUG }, 05:28:46 DEBUG "value": [ 05:28:46 DEBUG 1713760126, 05:28:46 DEBUG "339356071.0915524" 05:28:46 DEBUG ] 05:28:46 DEBUG } 05:28:46 DEBUG ] 05:28:46 DEBUG } 05:28:46 DEBUG } 05:28:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760127" 05:28:47 INFO [http_cmd]: http status code OK 05:28:47 DEBUG --- status code --- 05:28:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:47 DEBUG --- http response --- 05:28:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:47 DEBUG ----- output ----- 05:28:47 DEBUG { 05:28:47 DEBUG "status": "success", 05:28:47 DEBUG "data": { 05:28:47 DEBUG "resultType": "vector", 05:28:47 DEBUG "result": [ 05:28:47 DEBUG { 05:28:47 DEBUG "metric": { 05:28:47 DEBUG "instance": "10.142.0.107:9100" 05:28:47 DEBUG }, 05:28:47 DEBUG "value": [ 05:28:47 DEBUG 05:28:47 DEBUG 05:28:47 DEBUG 05:28:47 DEBUG [...] 05:28:47 DEBUG 05:28:47 DEBUG 05:28:47 DEBUG 05:28:47 DEBUG "instance": "10.142.0.43:9100" 05:28:47 DEBUG }, 05:28:47 DEBUG "value": [ 05:28:47 DEBUG 1713760127, 05:28:47 DEBUG "5789058550784" 05:28:47 DEBUG ] 05:28:47 DEBUG } 05:28:47 DEBUG ] 05:28:47 DEBUG } 05:28:47 DEBUG } 05:28:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760128" 05:28:48 INFO [http_cmd]: http status code OK 05:28:48 DEBUG --- status code --- 05:28:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:48 DEBUG --- http response --- 05:28:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:48 DEBUG ----- output ----- 05:28:48 DEBUG { 05:28:48 DEBUG "status": "success", 05:28:48 DEBUG "data": { 05:28:48 DEBUG "resultType": "vector", 05:28:48 DEBUG "result": [ 05:28:48 DEBUG { 05:28:48 DEBUG "metric": { 05:28:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:28:48 DEBUG }, 05:28:48 DEBUG "value": [ 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG [...] 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:28:48 DEBUG }, 05:28:48 DEBUG "value": [ 05:28:48 DEBUG 1713760128, 05:28:48 DEBUG "4249.133333333333" 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG } 05:28:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760128" 05:28:48 INFO [http_cmd]: http status code OK 05:28:48 DEBUG --- status code --- 05:28:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:48 DEBUG --- http response --- 05:28:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:28:48 DEBUG ----- output ----- 05:28:48 DEBUG { 05:28:48 DEBUG "status": "success", 05:28:48 DEBUG "data": { 05:28:48 DEBUG "resultType": "vector", 05:28:48 DEBUG "result": [ 05:28:48 DEBUG { 05:28:48 DEBUG "metric": { 05:28:48 DEBUG "pod": "ds-cts-0" 05:28:48 DEBUG }, 05:28:48 DEBUG "value": [ 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG [...] 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG "pod": "ds-cts-2" 05:28:48 DEBUG }, 05:28:48 DEBUG "value": [ 05:28:48 DEBUG 1713760128, 05:28:48 DEBUG "27.599999999999998" 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG } 05:28:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760128" 05:28:48 INFO [http_cmd]: http status code OK 05:28:48 DEBUG --- status code --- 05:28:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:48 DEBUG --- http response --- 05:28:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:48 DEBUG ----- output ----- 05:28:48 DEBUG { 05:28:48 DEBUG "status": "success", 05:28:48 DEBUG "data": { 05:28:48 DEBUG "resultType": "vector", 05:28:48 DEBUG "result": [ 05:28:48 DEBUG { 05:28:48 DEBUG "metric": { 05:28:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:28:48 DEBUG }, 05:28:48 DEBUG "value": [ 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG [...] 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:28:48 DEBUG }, 05:28:48 DEBUG "value": [ 05:28:48 DEBUG 1713760128, 05:28:48 DEBUG "955.1865" 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG } 05:28:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760128" 05:28:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760128" 05:28:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760128" 05:28:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760128" 05:28:48 INFO [http_cmd]: http status code OK 05:28:48 DEBUG --- status code --- 05:28:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:48 DEBUG --- http response --- 05:28:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:48 DEBUG ----- output ----- 05:28:48 DEBUG { 05:28:48 DEBUG "status": "success", 05:28:48 DEBUG "data": { 05:28:48 DEBUG "resultType": "vector", 05:28:48 DEBUG "result": [ 05:28:48 DEBUG { 05:28:48 DEBUG "metric": { 05:28:48 DEBUG "pod": "ds-idrepo-0" 05:28:48 DEBUG }, 05:28:48 DEBUG "value": [ 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG [...] 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG "pod": "ds-idrepo-2" 05:28:48 DEBUG }, 05:28:48 DEBUG "value": [ 05:28:48 DEBUG 1713760128, 05:28:48 DEBUG "0" 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG } 05:28:48 INFO [http_cmd]: http status code OK 05:28:48 DEBUG --- status code --- 05:28:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:48 INFO [http_cmd]: http status code OK 05:28:48 INFO [http_cmd]: http status code OK 05:28:48 DEBUG --- http response --- 05:28:48 DEBUG --- status code --- 05:28:48 DEBUG --- status code --- 05:28:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:48 DEBUG --- http response --- 05:28:48 DEBUG ----- output ----- 05:28:48 DEBUG --- http response --- 05:28:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:48 DEBUG { 05:28:48 DEBUG ----- output ----- 05:28:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:28:48 DEBUG "status": "success", 05:28:48 DEBUG { 05:28:48 DEBUG ----- output ----- 05:28:48 DEBUG "data": { 05:28:48 DEBUG "status": "success", 05:28:48 DEBUG { 05:28:48 DEBUG "resultType": "vector", 05:28:48 DEBUG "data": { 05:28:48 DEBUG "status": "success", 05:28:48 DEBUG "result": [ 05:28:48 DEBUG "resultType": "vector", 05:28:48 DEBUG "data": { 05:28:48 DEBUG { 05:28:48 DEBUG "result": [ 05:28:48 DEBUG "resultType": "vector", 05:28:48 DEBUG "metric": { 05:28:48 DEBUG { 05:28:48 DEBUG "result": [ 05:28:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:28:48 DEBUG "metric": { 05:28:48 DEBUG { 05:28:48 DEBUG }, 05:28:48 DEBUG "instance": "10.142.0.107:9100" 05:28:48 DEBUG "metric": { 05:28:48 DEBUG "value": [ 05:28:48 DEBUG }, 05:28:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:28:48 DEBUG 05:28:48 DEBUG "value": [ 05:28:48 DEBUG }, 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG "value": [ 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG [...] 05:28:48 DEBUG 05:28:48 DEBUG [...] 05:28:48 DEBUG 05:28:48 DEBUG [...] 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG 05:28:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:28:48 DEBUG 05:28:48 DEBUG "instance": "10.142.0.43:9100" 05:28:48 DEBUG }, 05:28:48 DEBUG "pod": "ds-idrepo-0" 05:28:48 DEBUG }, 05:28:48 DEBUG "value": [ 05:28:48 DEBUG 1713760128, 05:28:48 DEBUG "value": [ 05:28:48 DEBUG }, 05:28:48 DEBUG "564425.5" 05:28:48 DEBUG ] 05:28:48 DEBUG "value": [ 05:28:48 DEBUG } 05:28:48 DEBUG 1713760128, 05:28:48 DEBUG 1713760128, 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG } 05:28:48 DEBUG "0.44899999999996304" 05:28:48 DEBUG "5083.988415390981" 05:28:48 DEBUG ] 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG } 05:28:48 DEBUG ] 05:28:48 DEBUG ] 05:28:48 DEBUG } 05:28:48 DEBUG } 05:28:48 DEBUG } 05:28:48 DEBUG } 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760129" 05:28:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760129" 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 DEBUG --- status code --- 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG --- status code --- 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 DEBUG --- status code --- 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 INFO [http_cmd]: http status code OK 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG --- status code --- 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG --- http response --- 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG { 05:28:49 DEBUG { 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG { 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:49 DEBUG { 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG { 05:28:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:28:49 DEBUG { 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG { 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG { 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG { 05:28:49 DEBUG { 05:28:49 DEBUG { 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG ----- output ----- 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG { 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG "data": { 05:28:49 DEBUG "data": { 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG "data": { 05:28:49 DEBUG { 05:28:49 DEBUG "data": { 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG "data": { 05:28:49 DEBUG { 05:28:49 DEBUG "data": { 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG "data": { 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "data": { 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "data": { 05:28:49 DEBUG "data": { 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "data": { 05:28:49 DEBUG "status": "success", 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "data": { 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "result": [ 05:28:49 DEBUG "result": [ 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "result": [ 05:28:49 DEBUG "data": { 05:28:49 DEBUG "result": [ 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "result": [ 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "data": { 05:28:49 DEBUG "result": [ 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG "result": [ 05:28:49 DEBUG { 05:28:49 DEBUG { 05:28:49 DEBUG "result": [ 05:28:49 DEBUG { 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG { 05:28:49 DEBUG "result": [ 05:28:49 DEBUG "result": [ 05:28:49 DEBUG { 05:28:49 DEBUG "result": [ 05:28:49 DEBUG "resultType": "vector", 05:28:49 DEBUG { 05:28:49 DEBUG "result": [ 05:28:49 DEBUG { 05:28:49 DEBUG "metric": { 05:28:49 DEBUG "metric": { 05:28:49 DEBUG { 05:28:49 DEBUG "metric": { 05:28:49 DEBUG "result": [ 05:28:49 DEBUG "metric": { 05:28:49 DEBUG { 05:28:49 DEBUG { 05:28:49 DEBUG "metric": { 05:28:49 DEBUG { 05:28:49 DEBUG "result": [ 05:28:49 DEBUG "metric": { 05:28:49 DEBUG { 05:28:49 DEBUG "metric": { 05:28:49 DEBUG "pod": "ds-idrepo-0" 05:28:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:28:49 DEBUG "metric": { 05:28:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:28:49 DEBUG { 05:28:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:28:49 DEBUG "metric": { 05:28:49 DEBUG "metric": { 05:28:49 DEBUG "instance": "10.142.0.107:9100" 05:28:49 DEBUG "metric": { 05:28:49 DEBUG { 05:28:49 DEBUG "instance": "10.142.0.107:9100" 05:28:49 DEBUG "metric": { 05:28:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:28:49 DEBUG }, 05:28:49 DEBUG }, 05:28:49 DEBUG "instance": "10.142.0.107:9100" 05:28:49 DEBUG }, 05:28:49 DEBUG "metric": { 05:28:49 DEBUG }, 05:28:49 DEBUG "pod": "ds-cts-0" 05:28:49 DEBUG "pod": "ds-idrepo-0" 05:28:49 DEBUG }, 05:28:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:28:49 DEBUG "metric": { 05:28:49 DEBUG }, 05:28:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:28:49 DEBUG }, 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "value": [ 05:28:49 DEBUG }, 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:28:49 DEBUG "value": [ 05:28:49 DEBUG }, 05:28:49 DEBUG }, 05:28:49 DEBUG "value": [ 05:28:49 DEBUG }, 05:28:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:28:49 DEBUG "value": [ 05:28:49 DEBUG }, 05:28:49 DEBUG "value": [ 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG "value": [ 05:28:49 DEBUG 05:28:49 DEBUG }, 05:28:49 DEBUG 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "value": [ 05:28:49 DEBUG 05:28:49 DEBUG "value": [ 05:28:49 DEBUG }, 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG "value": [ 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG "value": [ 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG [...] 05:28:49 DEBUG [...] 05:28:49 DEBUG "value": [ 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG [...] 05:28:49 DEBUG [...] 05:28:49 DEBUG 05:28:49 DEBUG [...] 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG [...] 05:28:49 DEBUG [...] 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG [...] 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG [...] 05:28:49 DEBUG [...] 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG [...] 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG [...] 05:28:49 DEBUG [...] 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:28:49 DEBUG "pod": "ds-cts-2" 05:28:49 DEBUG [...] 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG "pod": "ds-idrepo-2" 05:28:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:28:49 DEBUG 05:28:49 DEBUG "pod": "ds-idrepo-0" 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG }, 05:28:49 DEBUG }, 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG "instance": "10.142.0.43:9100" 05:28:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:28:49 DEBUG 05:28:49 DEBUG }, 05:28:49 DEBUG }, 05:28:49 DEBUG "instance": "10.142.0.43:9100" 05:28:49 DEBUG }, 05:28:49 DEBUG 05:28:49 DEBUG 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "value": [ 05:28:49 DEBUG 05:28:49 DEBUG "instance": "10.142.0.43:9100" 05:28:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:28:49 DEBUG }, 05:28:49 DEBUG }, 05:28:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "value": [ 05:28:49 DEBUG }, 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:28:49 DEBUG "pod": "ds-cts-2" 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG 05:28:49 DEBUG }, 05:28:49 DEBUG }, 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "value": [ 05:28:49 DEBUG }, 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG "value": [ 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG }, 05:28:49 DEBUG }, 05:28:49 DEBUG "3.857013542832205" 05:28:49 DEBUG "0" 05:28:49 DEBUG "pod": "ds-idrepo-0" 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "value": [ 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "0" 05:28:49 DEBUG "0" 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG "3.1260615432372094" 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "value": [ 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG }, 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG "0.07833333333333789" 05:28:49 DEBUG "1087.1" 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG "24850951.43333333" 05:28:49 DEBUG ] 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG "value": [ 05:28:49 DEBUG "24856175.7" 05:28:49 DEBUG "3094536192" 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG "1120.6" 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG ] 05:28:49 DEBUG } 05:28:49 DEBUG "1119.5666666666666" 05:28:49 DEBUG "0" 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG 1713760129, 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG } 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG "0" 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG ] 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG ] 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG ] 05:28:49 DEBUG ] 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG ] 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:28:49 DEBUG } 05:29:06 INFO 05:29:06 INFO [loop_until]: kubectl --namespace=xlou top pods 05:29:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:29:06 INFO [loop_until]: OK (rc = 0) 05:29:06 DEBUG --- stdout --- 05:29:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 3700m 8881Mi am-767f6b67b5-qcqz2 3472m 8937Mi am-767f6b67b5-vbltz 3942m 8880Mi ds-cts-0 19349m 12519Mi ds-cts-1 18233m 12555Mi ds-cts-2 19535m 12560Mi ds-idrepo-0 911m 3993Mi ds-idrepo-1 923m 3981Mi ds-idrepo-2 899m 3964Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 11m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1m 2859Mi 05:29:06 DEBUG --- stderr --- 05:29:06 DEBUG 05:29:14 INFO 05:29:14 INFO [loop_until]: kubectl --namespace=xlou top node 05:29:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:29:14 INFO [loop_until]: OK (rc = 0) 05:29:14 DEBUG --- stdout --- 05:29:14 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6239Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 993m 3% 8094Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 3626m 12% 11682Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3341Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19158m 64% 15774Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5208Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 3652m 12% 11902Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3206Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 4266m 14% 11198Mi 10% gke-xlou-cdm-ds-0f601751-j488 988m 3% 7479Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18131m 60% 17738Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 1024m 3% 8040Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19222m 64% 15292Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 66m 0% 12820Mi 21% 05:29:14 DEBUG --- stderr --- 05:29:14 DEBUG 05:29:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760186" 05:29:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760186" 05:29:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760186" 05:29:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760186" 05:29:46 INFO [http_cmd]: http status code OK 05:29:46 INFO [http_cmd]: http status code OK 05:29:46 DEBUG --- status code --- 05:29:46 DEBUG --- status code --- 05:29:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:46 DEBUG --- http response --- 05:29:46 DEBUG --- http response --- 05:29:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:29:46 DEBUG ----- output ----- 05:29:46 DEBUG ----- output ----- 05:29:46 DEBUG { 05:29:46 DEBUG { 05:29:46 DEBUG "status": "success", 05:29:46 DEBUG "status": "success", 05:29:46 DEBUG "data": { 05:29:46 DEBUG "data": { 05:29:46 DEBUG "resultType": "vector", 05:29:46 DEBUG "resultType": "vector", 05:29:46 DEBUG "result": [ 05:29:46 DEBUG "result": [ 05:29:46 DEBUG { 05:29:46 DEBUG { 05:29:46 DEBUG "metric": { 05:29:46 DEBUG "metric": { 05:29:46 DEBUG "pod": "ds-cts-0" 05:29:46 DEBUG "pod": "ds-idrepo-0" 05:29:46 DEBUG }, 05:29:46 DEBUG }, 05:29:46 DEBUG "value": [ 05:29:46 DEBUG "value": [ 05:29:46 DEBUG 05:29:46 DEBUG 05:29:46 DEBUG 05:29:46 ERROR [http_cmd]: ERROR 05:29:46 DEBUG 05:29:46 DEBUG 05:29:46 ERROR --- status code --- 05:29:46 DEBUG 05:29:46 DEBUG [...] 05:29:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:29:46 DEBUG [...] 05:29:46 DEBUG 05:29:46 INFO [http_cmd]: http status code OK 05:29:46 ERROR --- http response --- 05:29:46 DEBUG 05:29:46 DEBUG 05:29:46 DEBUG --- status code --- 05:29:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:29:46 DEBUG 05:29:46 DEBUG 05:29:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:29:46 DEBUG 05:29:46 DEBUG "pod": "ds-cts-2" 05:29:46 DEBUG --- http response --- 05:29:46 DEBUG "pod": "ds-idrepo-2" 05:29:46 DEBUG }, 05:29:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:46 DEBUG }, 05:29:46 DEBUG "value": [ 05:29:46 DEBUG "value": [ 05:29:46 DEBUG 1713760186, 05:29:46 DEBUG 1713760186, 05:29:46 DEBUG "103.115" 05:29:46 DEBUG "0" 05:29:46 DEBUG ] 05:29:46 DEBUG ] 05:29:46 DEBUG } 05:29:46 DEBUG } 05:29:46 DEBUG ] 05:29:46 DEBUG ] 05:29:46 DEBUG } 05:29:46 DEBUG } 05:29:46 DEBUG } 05:29:46 DEBUG } 05:29:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760186" 05:29:46 INFO [http_cmd]: http status code OK 05:29:46 DEBUG --- status code --- 05:29:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:46 DEBUG --- http response --- 05:29:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:46 DEBUG ----- output ----- 05:29:46 DEBUG { 05:29:46 DEBUG "status": "success", 05:29:46 DEBUG "data": { 05:29:46 DEBUG "resultType": "vector", 05:29:46 DEBUG "result": [ 05:29:46 DEBUG { 05:29:46 DEBUG "metric": { 05:29:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:29:46 DEBUG }, 05:29:46 DEBUG "value": [ 05:29:46 DEBUG 05:29:46 DEBUG 05:29:46 DEBUG 05:29:46 DEBUG [...] 05:29:46 DEBUG 05:29:46 DEBUG 05:29:46 DEBUG 05:29:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:29:46 DEBUG }, 05:29:46 DEBUG "value": [ 05:29:46 DEBUG 1713760186, 05:29:46 DEBUG "33555919.955331154" 05:29:46 DEBUG ] 05:29:46 DEBUG } 05:29:46 DEBUG ] 05:29:46 DEBUG } 05:29:46 DEBUG } 05:29:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760187" 05:29:47 INFO [http_cmd]: http status code OK 05:29:47 DEBUG --- status code --- 05:29:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:47 DEBUG --- http response --- 05:29:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:47 DEBUG ----- output ----- 05:29:47 DEBUG { 05:29:47 DEBUG "status": "success", 05:29:47 DEBUG "data": { 05:29:47 DEBUG "resultType": "vector", 05:29:47 DEBUG "result": [ 05:29:47 DEBUG { 05:29:47 DEBUG "metric": { 05:29:47 DEBUG "instance": "10.142.0.107:9100" 05:29:47 DEBUG }, 05:29:47 DEBUG "value": [ 05:29:47 DEBUG 05:29:47 DEBUG 05:29:47 DEBUG 05:29:47 DEBUG [...] 05:29:47 DEBUG 05:29:47 DEBUG 05:29:47 DEBUG 05:29:47 DEBUG "instance": "10.142.0.43:9100" 05:29:47 DEBUG }, 05:29:47 DEBUG "value": [ 05:29:47 DEBUG 1713760187, 05:29:47 DEBUG "5792989381632" 05:29:47 DEBUG ] 05:29:47 DEBUG } 05:29:47 DEBUG ] 05:29:47 DEBUG } 05:29:47 DEBUG } 05:29:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760188" 05:29:48 INFO [http_cmd]: http status code OK 05:29:48 DEBUG --- status code --- 05:29:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:48 DEBUG --- http response --- 05:29:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:48 DEBUG ----- output ----- 05:29:48 DEBUG { 05:29:48 DEBUG "status": "success", 05:29:48 DEBUG "data": { 05:29:48 DEBUG "resultType": "vector", 05:29:48 DEBUG "result": [ 05:29:48 DEBUG { 05:29:48 DEBUG "metric": { 05:29:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:29:48 DEBUG }, 05:29:48 DEBUG "value": [ 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG [...] 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:29:48 DEBUG }, 05:29:48 DEBUG "value": [ 05:29:48 DEBUG 1713760188, 05:29:48 DEBUG "4293.7" 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG } 05:29:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760188" 05:29:48 INFO [http_cmd]: http status code OK 05:29:48 DEBUG --- status code --- 05:29:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:48 DEBUG --- http response --- 05:29:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:29:48 DEBUG ----- output ----- 05:29:48 DEBUG { 05:29:48 DEBUG "status": "success", 05:29:48 DEBUG "data": { 05:29:48 DEBUG "resultType": "vector", 05:29:48 DEBUG "result": [ 05:29:48 DEBUG { 05:29:48 DEBUG "metric": { 05:29:48 DEBUG "pod": "ds-cts-0" 05:29:48 DEBUG }, 05:29:48 DEBUG "value": [ 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG [...] 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG "pod": "ds-cts-2" 05:29:48 DEBUG }, 05:29:48 DEBUG "value": [ 05:29:48 DEBUG 1713760188, 05:29:48 DEBUG "30.333333333333332" 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG } 05:29:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760188" 05:29:48 INFO [http_cmd]: http status code OK 05:29:48 DEBUG --- status code --- 05:29:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:48 DEBUG --- http response --- 05:29:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:48 DEBUG ----- output ----- 05:29:48 DEBUG { 05:29:48 DEBUG "status": "success", 05:29:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760188" 05:29:48 DEBUG "data": { 05:29:48 DEBUG "resultType": "vector", 05:29:48 DEBUG "result": [ 05:29:48 DEBUG { 05:29:48 DEBUG "metric": { 05:29:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:29:48 DEBUG }, 05:29:48 DEBUG "value": [ 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG [...] 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:29:48 DEBUG }, 05:29:48 DEBUG "value": [ 05:29:48 DEBUG 1713760188, 05:29:48 DEBUG "959.05" 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG } 05:29:48 INFO [http_cmd]: http status code OK 05:29:48 DEBUG --- status code --- 05:29:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:48 DEBUG --- http response --- 05:29:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:48 DEBUG ----- output ----- 05:29:48 DEBUG { 05:29:48 DEBUG "status": "success", 05:29:48 DEBUG "data": { 05:29:48 DEBUG "resultType": "vector", 05:29:48 DEBUG "result": [ 05:29:48 DEBUG { 05:29:48 DEBUG "metric": { 05:29:48 DEBUG "pod": "ds-idrepo-0" 05:29:48 DEBUG }, 05:29:48 DEBUG "value": [ 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG [...] 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG "pod": "ds-idrepo-2" 05:29:48 DEBUG }, 05:29:48 DEBUG "value": [ 05:29:48 DEBUG 1713760188, 05:29:48 DEBUG "0" 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG } 05:29:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760188" 05:29:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760188" 05:29:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760188" 05:29:48 INFO [http_cmd]: http status code OK 05:29:48 DEBUG --- status code --- 05:29:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:48 INFO [http_cmd]: http status code OK 05:29:48 DEBUG --- http response --- 05:29:48 DEBUG --- status code --- 05:29:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:48 DEBUG ----- output ----- 05:29:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:48 INFO [http_cmd]: http status code OK 05:29:48 DEBUG { 05:29:48 DEBUG --- http response --- 05:29:48 DEBUG --- status code --- 05:29:48 DEBUG "status": "success", 05:29:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:48 DEBUG "data": { 05:29:48 DEBUG --- http response --- 05:29:48 DEBUG ----- output ----- 05:29:48 DEBUG "resultType": "vector", 05:29:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:48 DEBUG { 05:29:48 DEBUG "result": [ 05:29:48 DEBUG ----- output ----- 05:29:48 DEBUG "status": "success", 05:29:48 DEBUG { 05:29:48 DEBUG { 05:29:48 DEBUG "data": { 05:29:48 DEBUG "metric": { 05:29:48 DEBUG "status": "success", 05:29:48 DEBUG "resultType": "vector", 05:29:48 DEBUG "instance": "10.142.0.107:9100" 05:29:48 DEBUG "data": { 05:29:48 DEBUG "result": [ 05:29:48 DEBUG }, 05:29:48 DEBUG "resultType": "vector", 05:29:48 DEBUG { 05:29:48 DEBUG "value": [ 05:29:48 DEBUG "result": [ 05:29:48 DEBUG "metric": { 05:29:48 DEBUG 05:29:48 DEBUG { 05:29:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:29:48 DEBUG 05:29:48 DEBUG "metric": { 05:29:48 DEBUG }, 05:29:48 DEBUG 05:29:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:29:48 DEBUG "value": [ 05:29:48 DEBUG [...] 05:29:48 DEBUG }, 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG "value": [ 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG [...] 05:29:48 DEBUG "instance": "10.142.0.43:9100" 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG }, 05:29:48 DEBUG [...] 05:29:48 DEBUG 05:29:48 DEBUG "value": [ 05:29:48 DEBUG 05:29:48 DEBUG 05:29:48 DEBUG 1713760188, 05:29:48 DEBUG 05:29:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:29:48 DEBUG "0.5031000000000707" 05:29:48 DEBUG 05:29:48 DEBUG }, 05:29:48 DEBUG ] 05:29:48 DEBUG "pod": "ds-idrepo-0" 05:29:48 DEBUG }, 05:29:48 DEBUG } 05:29:48 DEBUG "value": [ 05:29:48 DEBUG "value": [ 05:29:48 DEBUG ] 05:29:48 DEBUG 1713760188, 05:29:48 DEBUG 1713760188, 05:29:48 DEBUG } 05:29:48 DEBUG "5074.173867261296" 05:29:48 DEBUG "383670.5" 05:29:48 DEBUG } 05:29:48 DEBUG ] 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG } 05:29:48 DEBUG ] 05:29:48 DEBUG } 05:29:48 DEBUG } 05:29:48 DEBUG } 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760189" 05:29:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760189" 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG --- status code --- 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG --- http response --- 05:29:49 INFO [http_cmd]: http status code OK 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG --- status code --- 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG --- http response --- 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:49 DEBUG { 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG ----- output ----- 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "data": { 05:29:49 DEBUG "data": { 05:29:49 DEBUG { 05:29:49 DEBUG "data": { 05:29:49 DEBUG "data": { 05:29:49 DEBUG "data": { 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "data": { 05:29:49 DEBUG "data": { 05:29:49 DEBUG "data": { 05:29:49 DEBUG { 05:29:49 DEBUG "data": { 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "data": { 05:29:49 DEBUG "data": { 05:29:49 DEBUG "data": { 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "status": "success", 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "data": { 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "data": { 05:29:49 DEBUG "result": [ 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "result": [ 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG "resultType": "vector", 05:29:49 DEBUG { 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "metric": { 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG { 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "result": [ 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "pod": "ds-idrepo-0" 05:29:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:29:49 DEBUG { 05:29:49 DEBUG "pod": "ds-idrepo-0" 05:29:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:29:49 DEBUG "instance": "10.142.0.107:9100" 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "metric": { 05:29:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:29:49 DEBUG "instance": "10.142.0.107:9100" 05:29:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:29:49 DEBUG { 05:29:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG "metric": { 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG "instance": "10.142.0.107:9100" 05:29:49 DEBUG "pod": "ds-cts-0" 05:29:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG "metric": { 05:29:49 DEBUG }, 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:29:49 DEBUG "value": [ 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG }, 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG }, 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG "value": [ 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG "value": [ 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG [...] 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG [...] 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG [...] 05:29:49 DEBUG [...] 05:29:49 DEBUG [...] 05:29:49 DEBUG [...] 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG [...] 05:29:49 DEBUG [...] 05:29:49 DEBUG [...] 05:29:49 DEBUG 05:29:49 DEBUG [...] 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG [...] 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG [...] 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG [...] 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG [...] 05:29:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG "pod": "ds-cts-2" 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG }, 05:29:49 DEBUG "pod": "ds-idrepo-2" 05:29:49 DEBUG "pod": "ds-idrepo-0" 05:29:49 DEBUG "instance": "10.142.0.43:9100" 05:29:49 DEBUG "instance": "10.142.0.43:9100" 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:29:49 DEBUG "instance": "10.142.0.43:9100" 05:29:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:29:49 DEBUG 05:29:49 DEBUG }, 05:29:49 DEBUG 05:29:49 DEBUG 05:29:49 DEBUG "value": [ 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG 05:29:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:29:49 DEBUG }, 05:29:49 DEBUG "value": [ 05:29:49 DEBUG }, 05:29:49 DEBUG "pod": "ds-idrepo-0" 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:29:49 DEBUG 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "pod": "ds-cts-2" 05:29:49 DEBUG }, 05:29:49 DEBUG }, 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG "value": [ 05:29:49 DEBUG }, 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG }, 05:29:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:29:49 DEBUG "0" 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG }, 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "0.02534642880099186" 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG "value": [ 05:29:49 DEBUG "0" 05:29:49 DEBUG "value": [ 05:29:49 DEBUG }, 05:29:49 DEBUG ] 05:29:49 DEBUG "0" 05:29:49 DEBUG "0.9044214128645688" 05:29:49 DEBUG "24696068.93333333" 05:29:49 DEBUG "24700174.9" 05:29:49 DEBUG "value": [ 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG ] 05:29:49 DEBUG "0" 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG ] 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG "value": [ 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG "0" 05:29:49 DEBUG "0.09899999999998954" 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG "0" 05:29:49 DEBUG } 05:29:49 DEBUG "3096907776" 05:29:49 DEBUG 1713760189, 05:29:49 DEBUG ] 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG "0" 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG "0" 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG ] 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG ] 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:29:49 DEBUG } 05:30:04 WARNING File /tmp/lodemon_data-20240421_231208/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [22/Apr/2024 05:30:04] "GET /monitoring/average?start_time=20240422_032918&stop_time=20240422_042803 HTTP/1.1" 200 - 05:30:06 INFO 05:30:06 INFO [loop_until]: kubectl --namespace=xlou top pods 05:30:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:30:06 INFO [loop_until]: OK (rc = 0) 05:30:06 DEBUG --- stdout --- 05:30:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 3538m 8810Mi am-767f6b67b5-qcqz2 3496m 8858Mi am-767f6b67b5-vbltz 3911m 8796Mi ds-cts-0 19003m 12512Mi ds-cts-1 18264m 12550Mi ds-cts-2 19969m 12561Mi ds-idrepo-0 909m 3993Mi ds-idrepo-1 923m 3981Mi ds-idrepo-2 892m 3964Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1181Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 6m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 3m 2859Mi 05:30:06 DEBUG --- stderr --- 05:30:06 DEBUG 05:30:14 INFO 05:30:14 INFO [loop_until]: kubectl --namespace=xlou top node 05:30:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:30:14 INFO [loop_until]: OK (rc = 0) 05:30:14 DEBUG --- stdout --- 05:30:14 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6237Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1006m 3% 8111Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 3593m 12% 11590Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3343Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 18750m 62% 15784Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5212Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 3629m 12% 11791Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3183Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 4090m 13% 11089Mi 9% gke-xlou-cdm-ds-0f601751-j488 984m 3% 7474Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18232m 61% 17721Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 1017m 3% 8057Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18370m 61% 15299Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 72m 0% 12822Mi 21% 05:30:14 DEBUG --- stderr --- 05:30:14 DEBUG 05:30:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760246" 05:30:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760246" 05:30:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760246" 05:30:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760246" 05:30:46 INFO [http_cmd]: http status code OK 05:30:46 ERROR [http_cmd]: ERROR 05:30:46 INFO [http_cmd]: http status code OK 05:30:46 DEBUG --- status code --- 05:30:46 ERROR --- status code --- 05:30:46 DEBUG --- status code --- 05:30:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:30:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:46 DEBUG --- http response --- 05:30:46 INFO [http_cmd]: http status code OK 05:30:46 ERROR --- http response --- 05:30:46 DEBUG --- http response --- 05:30:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:46 DEBUG --- status code --- 05:30:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:30:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:30:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:30:46 DEBUG ----- output ----- 05:30:46 DEBUG --- http response --- 05:30:46 DEBUG { 05:30:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:46 DEBUG "status": "success", 05:30:46 DEBUG ----- output ----- 05:30:46 DEBUG "data": { 05:30:46 DEBUG { 05:30:46 DEBUG "resultType": "vector", 05:30:46 DEBUG "status": "success", 05:30:46 DEBUG "result": [ 05:30:46 DEBUG "data": { 05:30:46 DEBUG { 05:30:46 DEBUG "resultType": "vector", 05:30:46 DEBUG "metric": { 05:30:46 DEBUG "result": [ 05:30:46 DEBUG "pod": "ds-cts-0" 05:30:46 DEBUG { 05:30:46 DEBUG }, 05:30:46 DEBUG "metric": { 05:30:46 DEBUG "value": [ 05:30:46 DEBUG "pod": "ds-idrepo-0" 05:30:46 DEBUG 05:30:46 DEBUG }, 05:30:46 DEBUG 05:30:46 DEBUG "value": [ 05:30:46 DEBUG 05:30:46 DEBUG 05:30:46 DEBUG [...] 05:30:46 DEBUG 05:30:46 DEBUG 05:30:46 DEBUG 05:30:46 DEBUG "pod": "ds-cts-2" 05:30:46 DEBUG }, 05:30:46 DEBUG 05:30:46 DEBUG 05:30:46 DEBUG [...] 05:30:46 DEBUG 05:30:46 DEBUG 05:30:46 DEBUG "value": [ 05:30:46 DEBUG 05:30:46 DEBUG 1713760246, 05:30:46 DEBUG "pod": "ds-idrepo-2" 05:30:46 DEBUG "102.6597" 05:30:46 DEBUG }, 05:30:46 DEBUG ] 05:30:46 DEBUG "value": [ 05:30:46 DEBUG } 05:30:46 DEBUG 1713760246, 05:30:46 DEBUG ] 05:30:46 DEBUG "0.06666666666666667" 05:30:46 DEBUG } 05:30:46 DEBUG ] 05:30:46 DEBUG } 05:30:46 DEBUG ] 05:30:46 DEBUG } 05:30:46 DEBUG } 05:30:46 DEBUG } 05:30:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760246" 05:30:46 INFO [http_cmd]: http status code OK 05:30:46 DEBUG --- status code --- 05:30:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:46 DEBUG --- http response --- 05:30:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:46 DEBUG ----- output ----- 05:30:46 DEBUG { 05:30:46 DEBUG "status": "success", 05:30:46 DEBUG "data": { 05:30:46 DEBUG "resultType": "vector", 05:30:46 DEBUG "result": [ 05:30:46 DEBUG { 05:30:46 DEBUG "metric": { 05:30:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:30:46 DEBUG }, 05:30:46 DEBUG "value": [ 05:30:46 DEBUG 05:30:46 DEBUG 05:30:46 DEBUG 05:30:46 DEBUG [...] 05:30:46 DEBUG 05:30:46 DEBUG 05:30:46 DEBUG 05:30:46 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:30:46 DEBUG }, 05:30:46 DEBUG "value": [ 05:30:46 DEBUG 1713760246, 05:30:46 DEBUG "197822704.5713131" 05:30:46 DEBUG ] 05:30:46 DEBUG } 05:30:46 DEBUG ] 05:30:46 DEBUG } 05:30:46 DEBUG } 05:30:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760247" 05:30:47 INFO [http_cmd]: http status code OK 05:30:47 DEBUG --- status code --- 05:30:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:47 DEBUG --- http response --- 05:30:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:47 DEBUG ----- output ----- 05:30:47 DEBUG { 05:30:47 DEBUG "status": "success", 05:30:47 DEBUG "data": { 05:30:47 DEBUG "resultType": "vector", 05:30:47 DEBUG "result": [ 05:30:47 DEBUG { 05:30:47 DEBUG "metric": { 05:30:47 DEBUG "instance": "10.142.0.107:9100" 05:30:47 DEBUG }, 05:30:47 DEBUG "value": [ 05:30:47 DEBUG 05:30:47 DEBUG 05:30:47 DEBUG 05:30:47 DEBUG [...] 05:30:47 DEBUG 05:30:47 DEBUG 05:30:47 DEBUG 05:30:47 DEBUG "instance": "10.142.0.43:9100" 05:30:47 DEBUG }, 05:30:47 DEBUG "value": [ 05:30:47 DEBUG 1713760247, 05:30:47 DEBUG "5796738731008" 05:30:47 DEBUG ] 05:30:47 DEBUG } 05:30:47 DEBUG ] 05:30:47 DEBUG } 05:30:47 DEBUG } 05:30:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760248" 05:30:48 INFO [http_cmd]: http status code OK 05:30:48 DEBUG --- status code --- 05:30:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:48 DEBUG --- http response --- 05:30:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:48 DEBUG ----- output ----- 05:30:48 DEBUG { 05:30:48 DEBUG "status": "success", 05:30:48 DEBUG "data": { 05:30:48 DEBUG "resultType": "vector", 05:30:48 DEBUG "result": [ 05:30:48 DEBUG { 05:30:48 DEBUG "metric": { 05:30:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:30:48 DEBUG }, 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG [...] 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:30:48 DEBUG }, 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 1713760248, 05:30:48 DEBUG "4224.166666666666" 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG } 05:30:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760248" 05:30:48 INFO [http_cmd]: http status code OK 05:30:48 DEBUG --- status code --- 05:30:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:48 DEBUG --- http response --- 05:30:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:30:48 DEBUG ----- output ----- 05:30:48 DEBUG { 05:30:48 DEBUG "status": "success", 05:30:48 DEBUG "data": { 05:30:48 DEBUG "resultType": "vector", 05:30:48 DEBUG "result": [ 05:30:48 DEBUG { 05:30:48 DEBUG "metric": { 05:30:48 DEBUG "pod": "ds-cts-0" 05:30:48 DEBUG }, 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG [...] 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG "pod": "ds-cts-2" 05:30:48 DEBUG }, 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 1713760248, 05:30:48 DEBUG "29.666666666666668" 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG } 05:30:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760248" 05:30:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760248" 05:30:48 INFO [http_cmd]: http status code OK 05:30:48 DEBUG --- status code --- 05:30:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:48 DEBUG --- http response --- 05:30:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:48 DEBUG ----- output ----- 05:30:48 DEBUG { 05:30:48 DEBUG "status": "success", 05:30:48 DEBUG "data": { 05:30:48 DEBUG "resultType": "vector", 05:30:48 DEBUG "result": [ 05:30:48 DEBUG { 05:30:48 DEBUG "metric": { 05:30:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:30:48 DEBUG }, 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG [...] 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:30:48 DEBUG }, 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 1713760248, 05:30:48 DEBUG "962.2339999999999" 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG } 05:30:48 INFO [http_cmd]: http status code OK 05:30:48 DEBUG --- status code --- 05:30:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:48 DEBUG --- http response --- 05:30:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:48 DEBUG ----- output ----- 05:30:48 DEBUG { 05:30:48 DEBUG "status": "success", 05:30:48 DEBUG "data": { 05:30:48 DEBUG "resultType": "vector", 05:30:48 DEBUG "result": [ 05:30:48 DEBUG { 05:30:48 DEBUG "metric": { 05:30:48 DEBUG "pod": "ds-idrepo-0" 05:30:48 DEBUG }, 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG [...] 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG "pod": "ds-idrepo-2" 05:30:48 DEBUG }, 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 1713760248, 05:30:48 DEBUG "0.06666666666666667" 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG } 05:30:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760248" 05:30:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760248" 05:30:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760248" 05:30:48 INFO [http_cmd]: http status code OK 05:30:48 DEBUG --- status code --- 05:30:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:48 DEBUG --- http response --- 05:30:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:48 DEBUG ----- output ----- 05:30:48 DEBUG { 05:30:48 DEBUG "status": "success", 05:30:48 DEBUG "data": { 05:30:48 DEBUG "resultType": "vector", 05:30:48 DEBUG "result": [ 05:30:48 DEBUG { 05:30:48 DEBUG "metric": { 05:30:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:30:48 DEBUG }, 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG [...] 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG "pod": "ds-cts-0" 05:30:48 DEBUG }, 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 1713760248, 05:30:48 DEBUG "58609968.47800517" 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG } 05:30:48 INFO [http_cmd]: http status code OK 05:30:48 DEBUG --- status code --- 05:30:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:48 DEBUG --- http response --- 05:30:48 INFO [http_cmd]: http status code OK 05:30:48 DEBUG --- status code --- 05:30:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:48 DEBUG ----- output ----- 05:30:48 DEBUG --- http response --- 05:30:48 DEBUG { 05:30:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:48 DEBUG "status": "success", 05:30:48 DEBUG ----- output ----- 05:30:48 DEBUG "data": { 05:30:48 DEBUG { 05:30:48 DEBUG "resultType": "vector", 05:30:48 DEBUG "status": "success", 05:30:48 DEBUG "result": [ 05:30:48 DEBUG "data": { 05:30:48 DEBUG { 05:30:48 DEBUG "resultType": "vector", 05:30:48 DEBUG "metric": { 05:30:48 DEBUG "result": [ 05:30:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:30:48 DEBUG { 05:30:48 DEBUG }, 05:30:48 DEBUG "metric": { 05:30:48 DEBUG "value": [ 05:30:48 DEBUG "instance": "10.142.0.107:9100" 05:30:48 DEBUG 05:30:48 DEBUG }, 05:30:48 DEBUG 05:30:48 DEBUG "value": [ 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG [...] 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG [...] 05:30:48 DEBUG 05:30:48 DEBUG 05:30:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:30:48 DEBUG 05:30:48 DEBUG }, 05:30:48 DEBUG 05:30:48 DEBUG "value": [ 05:30:48 DEBUG "instance": "10.142.0.43:9100" 05:30:48 DEBUG 1713760248, 05:30:48 DEBUG }, 05:30:48 DEBUG "185717" 05:30:48 DEBUG "value": [ 05:30:48 DEBUG ] 05:30:48 DEBUG 1713760248, 05:30:48 DEBUG } 05:30:48 DEBUG "0.4886333333334562" 05:30:48 DEBUG ] 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG } 05:30:48 DEBUG } 05:30:48 DEBUG ] 05:30:48 DEBUG } 05:30:48 DEBUG } 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760249" 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG { 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG "data": { 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "result": [ 05:30:49 DEBUG { 05:30:49 DEBUG "metric": { 05:30:49 DEBUG "instance": "10.142.0.107:9100" 05:30:49 DEBUG }, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 DEBUG 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG "instance": "10.142.0.43:9100" 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG { 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG "data": { 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "result": [ 05:30:49 DEBUG { 05:30:49 DEBUG "metric": { 05:30:49 DEBUG "instance": "10.142.0.107:9100" 05:30:49 DEBUG }, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG "instance": "10.142.0.43:9100" 05:30:49 DEBUG }, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG "0.08366666666668343" 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG }, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG "25405053.333333332" 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760249" 05:30:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760249" 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 DEBUG --- status code --- 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG --- status code --- 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG --- http response --- 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:49 DEBUG --- status code --- 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 INFO [http_cmd]: http status code OK 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG --- status code --- 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG { 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:49 DEBUG { 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:49 DEBUG { 05:30:49 DEBUG { 05:30:49 DEBUG { 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG { 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG { 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG { 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG { 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG { 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG "data": { 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG { 05:30:49 DEBUG --- http response --- 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG "data": { 05:30:49 DEBUG "data": { 05:30:49 DEBUG "data": { 05:30:49 DEBUG "data": { 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG "data": { 05:30:49 DEBUG "data": { 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "data": { 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:30:49 DEBUG ----- output ----- 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "data": { 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "result": [ 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "data": { 05:30:49 DEBUG { 05:30:49 DEBUG "data": { 05:30:49 DEBUG "result": [ 05:30:49 DEBUG "result": [ 05:30:49 DEBUG "result": [ 05:30:49 DEBUG "result": [ 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "result": [ 05:30:49 DEBUG "result": [ 05:30:49 DEBUG { 05:30:49 DEBUG "result": [ 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG "status": "success", 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG { 05:30:49 DEBUG { 05:30:49 DEBUG { 05:30:49 DEBUG { 05:30:49 DEBUG "result": [ 05:30:49 DEBUG { 05:30:49 DEBUG { 05:30:49 DEBUG "metric": { 05:30:49 DEBUG { 05:30:49 DEBUG "result": [ 05:30:49 DEBUG "data": { 05:30:49 DEBUG "result": [ 05:30:49 DEBUG "metric": { 05:30:49 DEBUG "metric": { 05:30:49 DEBUG "metric": { 05:30:49 DEBUG "metric": { 05:30:49 DEBUG { 05:30:49 DEBUG "metric": { 05:30:49 DEBUG "metric": { 05:30:49 DEBUG "pod": "ds-idrepo-0" 05:30:49 DEBUG "metric": { 05:30:49 DEBUG { 05:30:49 DEBUG "resultType": "vector", 05:30:49 DEBUG { 05:30:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:30:49 DEBUG "pod": "ds-idrepo-0" 05:30:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:30:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:30:49 DEBUG "metric": { 05:30:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:30:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:30:49 DEBUG }, 05:30:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:30:49 DEBUG "metric": { 05:30:49 DEBUG "instance": "10.142.0.107:9100" 05:30:49 DEBUG "result": [ 05:30:49 DEBUG }, 05:30:49 DEBUG }, 05:30:49 DEBUG }, 05:30:49 DEBUG }, 05:30:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:30:49 DEBUG }, 05:30:49 DEBUG }, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG }, 05:30:49 DEBUG }, 05:30:49 DEBUG "metric": { 05:30:49 DEBUG { 05:30:49 DEBUG "value": [ 05:30:49 DEBUG "value": [ 05:30:49 DEBUG "value": [ 05:30:49 DEBUG "value": [ 05:30:49 DEBUG }, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 05:30:49 DEBUG "value": [ 05:30:49 DEBUG "value": [ 05:30:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:30:49 DEBUG "metric": { 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG }, 05:30:49 DEBUG "pod": "ds-cts-0" 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG "value": [ 05:30:49 DEBUG }, 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG "value": [ 05:30:49 DEBUG [...] 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 DEBUG [...] 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 DEBUG [...] 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG "pod": "ds-cts-2" 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG "pod": "ds-idrepo-2" 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG [...] 05:30:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:30:49 DEBUG }, 05:30:49 DEBUG 05:30:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:30:49 DEBUG 05:30:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:30:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:30:49 DEBUG }, 05:30:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:30:49 DEBUG "instance": "10.142.0.43:9100" 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG }, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG "pod": "ds-cts-0" 05:30:49 DEBUG }, 05:30:49 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:30:49 DEBUG }, 05:30:49 DEBUG }, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG }, 05:30:49 DEBUG }, 05:30:49 DEBUG 05:30:49 DEBUG 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG }, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG }, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG "value": [ 05:30:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:30:49 DEBUG 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG "0" 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG "0.06666666666666667" 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG }, 05:30:49 DEBUG "pod": "ds-cts-2" 05:30:49 DEBUG "0.030580543839043607" 05:30:49 DEBUG ] 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG "0" 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG "0" 05:30:49 DEBUG "0" 05:30:49 DEBUG ] 05:30:49 DEBUG "0" 05:30:49 DEBUG "25239080.5" 05:30:49 DEBUG "value": [ 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG "0" 05:30:49 DEBUG ] 05:30:49 DEBUG "0" 05:30:49 DEBUG ] 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG ] 05:30:49 DEBUG "3097014272" 05:30:49 DEBUG }, 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG "value": [ 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG ] 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG 1713760249, 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG "0" 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG } 05:30:49 DEBUG ] 05:30:49 DEBUG } 05:30:49 DEBUG } 05:31:06 INFO 05:31:06 INFO [loop_until]: kubectl --namespace=xlou top pods 05:31:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:31:06 INFO [loop_until]: OK (rc = 0) 05:31:06 DEBUG --- stdout --- 05:31:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 3729m 8811Mi am-767f6b67b5-qcqz2 3634m 8866Mi am-767f6b67b5-vbltz 3786m 8796Mi ds-cts-0 18890m 12507Mi ds-cts-1 18621m 12546Mi ds-cts-2 19213m 12553Mi ds-idrepo-0 1282m 3999Mi ds-idrepo-1 1224m 3981Mi ds-idrepo-2 1169m 3976Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 1183Mi idm-b85f9d978-wnhxb 5m 1244Mi lodemon-7f68c498d9-gbc5j 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 531m 3196Mi 05:31:06 DEBUG --- stderr --- 05:31:06 DEBUG 05:31:14 INFO 05:31:14 INFO [loop_until]: kubectl --namespace=xlou top node 05:31:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:31:14 INFO [loop_until]: OK (rc = 0) 05:31:14 DEBUG --- stdout --- 05:31:14 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6237Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1272m 4% 8131Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 3657m 12% 11581Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3365Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18996m 63% 15760Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5197Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 3705m 12% 11809Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 87m 0% 3162Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 4219m 14% 11080Mi 9% gke-xlou-cdm-ds-0f601751-j488 1249m 4% 7462Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18712m 62% 17726Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 1245m 4% 8029Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19280m 64% 15260Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 858m 5% 13178Mi 22% 05:31:14 DEBUG --- stderr --- 05:31:14 DEBUG 05:31:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760306" 05:31:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760306" 05:31:46 ERROR [http_cmd]: ERROR 05:31:46 ERROR --- status code --- 05:31:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:31:46 ERROR --- http response --- 05:31:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760306" 05:31:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760306" 05:31:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:31:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:31:46 INFO [http_cmd]: http status code OK 05:31:46 DEBUG --- status code --- 05:31:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:46 DEBUG --- http response --- 05:31:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760306" 05:31:46 INFO [http_cmd]: http status code OK 05:31:46 INFO [http_cmd]: http status code OK 05:31:46 DEBUG --- status code --- 05:31:46 DEBUG --- status code --- 05:31:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:46 DEBUG --- http response --- 05:31:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:46 DEBUG --- http response --- 05:31:46 DEBUG ----- output ----- 05:31:46 DEBUG { 05:31:46 DEBUG "status": "success", 05:31:46 DEBUG "data": { 05:31:46 DEBUG "resultType": "vector", 05:31:46 DEBUG "result": [ 05:31:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:31:46 DEBUG { 05:31:46 INFO [http_cmd]: http status code OK 05:31:46 DEBUG ----- output ----- 05:31:46 DEBUG "metric": { 05:31:46 DEBUG --- status code --- 05:31:46 DEBUG { 05:31:46 DEBUG "pod": "ds-idrepo-0" 05:31:46 DEBUG }, 05:31:46 DEBUG "status": "success", 05:31:46 DEBUG "value": [ 05:31:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:46 DEBUG "data": { 05:31:46 DEBUG 05:31:46 DEBUG --- http response --- 05:31:46 DEBUG "resultType": "vector", 05:31:46 DEBUG 05:31:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:31:46 DEBUG "result": [ 05:31:46 DEBUG 05:31:46 DEBUG ----- output ----- 05:31:46 DEBUG { 05:31:46 DEBUG [...] 05:31:46 DEBUG { 05:31:46 DEBUG "metric": { 05:31:46 DEBUG 05:31:46 DEBUG "status": "success", 05:31:46 DEBUG "pod": "ds-cts-0" 05:31:46 DEBUG 05:31:46 DEBUG "data": { 05:31:46 DEBUG }, 05:31:46 DEBUG 05:31:46 DEBUG "resultType": "vector", 05:31:46 DEBUG "value": [ 05:31:46 DEBUG "pod": "ds-idrepo-2" 05:31:46 DEBUG "result": [ 05:31:46 DEBUG 05:31:46 DEBUG }, 05:31:46 DEBUG { 05:31:46 DEBUG 05:31:46 DEBUG "value": [ 05:31:46 DEBUG "metric": { 05:31:46 DEBUG 05:31:46 DEBUG [...] 05:31:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:31:46 DEBUG }, 05:31:46 DEBUG 05:31:46 DEBUG 1713760306, 05:31:46 DEBUG "value": [ 05:31:46 DEBUG 05:31:46 DEBUG "0.03333333333333333" 05:31:46 DEBUG 05:31:46 DEBUG 05:31:46 DEBUG ] 05:31:46 DEBUG 05:31:46 DEBUG "pod": "ds-cts-2" 05:31:46 DEBUG } 05:31:46 DEBUG 05:31:46 DEBUG }, 05:31:46 DEBUG ] 05:31:46 DEBUG [...] 05:31:46 DEBUG "value": [ 05:31:46 DEBUG } 05:31:46 DEBUG 05:31:46 DEBUG 1713760306, 05:31:46 DEBUG } 05:31:46 DEBUG 05:31:46 DEBUG "101.50526666666666" 05:31:46 DEBUG 05:31:46 DEBUG ] 05:31:46 DEBUG "pod": "ds-cts-0" 05:31:46 DEBUG } 05:31:46 DEBUG }, 05:31:46 DEBUG ] 05:31:46 DEBUG "value": [ 05:31:46 DEBUG } 05:31:46 DEBUG 1713760306, 05:31:46 DEBUG } 05:31:46 DEBUG "42485.67609971526" 05:31:46 DEBUG ] 05:31:46 DEBUG } 05:31:46 DEBUG ] 05:31:46 DEBUG } 05:31:46 DEBUG } 05:31:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760307" 05:31:47 INFO [http_cmd]: http status code OK 05:31:47 DEBUG --- status code --- 05:31:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:47 DEBUG --- http response --- 05:31:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:47 DEBUG ----- output ----- 05:31:47 DEBUG { 05:31:47 DEBUG "status": "success", 05:31:47 DEBUG "data": { 05:31:47 DEBUG "resultType": "vector", 05:31:47 DEBUG "result": [ 05:31:47 DEBUG { 05:31:47 DEBUG "metric": { 05:31:47 DEBUG "instance": "10.142.0.107:9100" 05:31:47 DEBUG }, 05:31:47 DEBUG "value": [ 05:31:47 DEBUG 05:31:47 DEBUG 05:31:47 DEBUG 05:31:47 DEBUG [...] 05:31:47 DEBUG 05:31:47 DEBUG 05:31:47 DEBUG 05:31:47 DEBUG "instance": "10.142.0.43:9100" 05:31:47 DEBUG }, 05:31:47 DEBUG "value": [ 05:31:47 DEBUG 1713760307, 05:31:47 DEBUG "5800001385472" 05:31:47 DEBUG ] 05:31:47 DEBUG } 05:31:47 DEBUG ] 05:31:47 DEBUG } 05:31:47 DEBUG } 05:31:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760308" 05:31:48 INFO [http_cmd]: http status code OK 05:31:48 DEBUG --- status code --- 05:31:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:48 DEBUG --- http response --- 05:31:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:48 DEBUG ----- output ----- 05:31:48 DEBUG { 05:31:48 DEBUG "status": "success", 05:31:48 DEBUG "data": { 05:31:48 DEBUG "resultType": "vector", 05:31:48 DEBUG "result": [ 05:31:48 DEBUG { 05:31:48 DEBUG "metric": { 05:31:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:31:48 DEBUG }, 05:31:48 DEBUG "value": [ 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG [...] 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:31:48 DEBUG }, 05:31:48 DEBUG "value": [ 05:31:48 DEBUG 1713760308, 05:31:48 DEBUG "2222.1" 05:31:48 DEBUG ] 05:31:48 DEBUG } 05:31:48 DEBUG ] 05:31:48 DEBUG } 05:31:48 DEBUG } 05:31:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760308" 05:31:48 INFO [http_cmd]: http status code OK 05:31:48 DEBUG --- status code --- 05:31:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:48 DEBUG --- http response --- 05:31:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:31:48 DEBUG ----- output ----- 05:31:48 DEBUG { 05:31:48 DEBUG "status": "success", 05:31:48 DEBUG "data": { 05:31:48 DEBUG "resultType": "vector", 05:31:48 DEBUG "result": [ 05:31:48 DEBUG { 05:31:48 DEBUG "metric": { 05:31:48 DEBUG "pod": "ds-cts-0" 05:31:48 DEBUG }, 05:31:48 DEBUG "value": [ 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG [...] 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG "pod": "ds-cts-2" 05:31:48 DEBUG }, 05:31:48 DEBUG "value": [ 05:31:48 DEBUG 1713760308, 05:31:48 DEBUG "33.666666666666664" 05:31:48 DEBUG ] 05:31:48 DEBUG } 05:31:48 DEBUG ] 05:31:48 DEBUG } 05:31:48 DEBUG } 05:31:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760308" 05:31:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760308" 05:31:48 INFO [http_cmd]: http status code OK 05:31:48 DEBUG --- status code --- 05:31:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:48 DEBUG --- http response --- 05:31:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:48 DEBUG ----- output ----- 05:31:48 DEBUG { 05:31:48 DEBUG "status": "success", 05:31:48 DEBUG "data": { 05:31:48 DEBUG "resultType": "vector", 05:31:48 DEBUG "result": [ 05:31:48 DEBUG { 05:31:48 DEBUG "metric": { 05:31:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:31:48 DEBUG }, 05:31:48 DEBUG "value": [ 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG [...] 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:31:48 DEBUG }, 05:31:48 DEBUG "value": [ 05:31:48 DEBUG 1713760308, 05:31:48 DEBUG "965.2280000000001" 05:31:48 DEBUG ] 05:31:48 DEBUG } 05:31:48 DEBUG ] 05:31:48 DEBUG } 05:31:48 DEBUG } 05:31:48 INFO [http_cmd]: http status code OK 05:31:48 DEBUG --- status code --- 05:31:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:48 DEBUG --- http response --- 05:31:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:48 DEBUG ----- output ----- 05:31:48 DEBUG { 05:31:48 DEBUG "status": "success", 05:31:48 DEBUG "data": { 05:31:48 DEBUG "resultType": "vector", 05:31:48 DEBUG "result": [ 05:31:48 DEBUG { 05:31:48 DEBUG "metric": { 05:31:48 DEBUG "pod": "ds-idrepo-0" 05:31:48 DEBUG }, 05:31:48 DEBUG "value": [ 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG [...] 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG 05:31:48 DEBUG "pod": "ds-idrepo-2" 05:31:48 DEBUG }, 05:31:48 DEBUG "value": [ 05:31:48 DEBUG 1713760308, 05:31:48 DEBUG "0.03333333333333333" 05:31:48 DEBUG ] 05:31:48 DEBUG } 05:31:48 DEBUG ] 05:31:48 DEBUG } 05:31:48 DEBUG } 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG { 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "data": { 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "result": [ 05:31:49 DEBUG { 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:31:49 DEBUG }, 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG [...] 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG "pod": "ds-cts-0" 05:31:49 DEBUG }, 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 1713760309, 05:31:49 DEBUG "60630868.03416816" 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG ] 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG } 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG ] 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:49 DEBUG } 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG { 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG { 05:31:49 DEBUG } 05:31:49 DEBUG "data": { 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "data": { 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG { 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "metric": { 05:31:49 DEBUG { 05:31:49 DEBUG "instance": "10.142.0.107:9100" 05:31:49 DEBUG "metric": { 05:31:49 DEBUG }, 05:31:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:31:49 DEBUG "value": [ 05:31:49 DEBUG }, 05:31:49 DEBUG 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG [...] 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG [...] 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG "instance": "10.142.0.43:9100" 05:31:49 DEBUG 05:31:49 DEBUG }, 05:31:49 DEBUG 05:31:49 DEBUG "value": [ 05:31:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:31:49 DEBUG 1713760309, 05:31:49 DEBUG }, 05:31:49 DEBUG "0.5072666666666105" 05:31:49 DEBUG "value": [ 05:31:49 DEBUG ] 05:31:49 DEBUG 1713760309, 05:31:49 DEBUG } 05:31:49 DEBUG "18492" 05:31:49 DEBUG ] 05:31:49 DEBUG ] 05:31:49 DEBUG } 05:31:49 DEBUG } 05:31:49 DEBUG } 05:31:49 DEBUG ] 05:31:49 DEBUG } 05:31:49 DEBUG } 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760309" 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG { 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "data": { 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "result": [ 05:31:49 DEBUG { 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "instance": "10.142.0.107:9100" 05:31:49 DEBUG }, 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG [...] 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG "instance": "10.142.0.43:9100" 05:31:49 DEBUG }, 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 1713760309, 05:31:49 DEBUG "4917152.766666668" 05:31:49 DEBUG ] 05:31:49 DEBUG } 05:31:49 DEBUG ] 05:31:49 DEBUG } 05:31:49 DEBUG } 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG { 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "data": { 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "result": [ 05:31:49 DEBUG { 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "instance": "10.142.0.107:9100" 05:31:49 DEBUG }, 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG [...] 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG "instance": "10.142.0.43:9100" 05:31:49 DEBUG }, 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 1713760309, 05:31:49 DEBUG "0.07333333333334051" 05:31:49 DEBUG ] 05:31:49 DEBUG } 05:31:49 DEBUG ] 05:31:49 DEBUG } 05:31:49 DEBUG } 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760309" 05:31:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760309" 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG --- status code --- 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 DEBUG --- status code --- 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 DEBUG --- status code --- 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 DEBUG --- status code --- 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:49 INFO [http_cmd]: http status code OK 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG { 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG --- status code --- 05:31:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG { 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG { 05:31:49 DEBUG { 05:31:49 DEBUG --- http response --- 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG { 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG { 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG "data": { 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:49 DEBUG { 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG { 05:31:49 DEBUG { 05:31:49 DEBUG { 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "data": { 05:31:49 DEBUG { 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "data": { 05:31:49 DEBUG "data": { 05:31:49 DEBUG ----- output ----- 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "data": { 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "data": { 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG { 05:31:49 DEBUG "data": { 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "data": { 05:31:49 DEBUG "data": { 05:31:49 DEBUG "data": { 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "data": { 05:31:49 DEBUG { 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "status": "success", 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "result": [ 05:31:49 DEBUG { 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG "metric": { 05:31:49 DEBUG { 05:31:49 DEBUG { 05:31:49 DEBUG "data": { 05:31:49 DEBUG "result": [ 05:31:49 DEBUG { 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "result": [ 05:31:49 DEBUG { 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "resultType": "vector", 05:31:49 DEBUG { 05:31:49 DEBUG "metric": { 05:31:49 DEBUG { 05:31:49 DEBUG { 05:31:49 DEBUG { 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "pod": "ds-idrepo-0" 05:31:49 DEBUG { 05:31:49 DEBUG }, 05:31:49 DEBUG "pod": "ds-idrepo-0" 05:31:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:31:49 DEBUG }, 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:31:49 DEBUG }, 05:31:49 DEBUG "metric": { 05:31:49 DEBUG "value": [ 05:31:49 DEBUG }, 05:31:49 DEBUG "result": [ 05:31:49 DEBUG "value": [ 05:31:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:31:49 DEBUG }, 05:31:49 DEBUG "instance": "10.142.0.107:9100" 05:31:49 DEBUG "pod": "ds-cts-0" 05:31:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:31:49 DEBUG }, 05:31:49 DEBUG "value": [ 05:31:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:31:49 DEBUG 05:31:49 DEBUG "value": [ 05:31:49 DEBUG { 05:31:49 DEBUG 05:31:49 DEBUG }, 05:31:49 DEBUG "value": [ 05:31:49 DEBUG }, 05:31:49 DEBUG }, 05:31:49 DEBUG }, 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 05:31:49 DEBUG }, 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG "metric": { 05:31:49 DEBUG 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 05:31:49 DEBUG "value": [ 05:31:49 DEBUG "value": [ 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG "value": [ 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:49 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG [...] 05:31:50 DEBUG 05:31:50 DEBUG }, 05:31:50 DEBUG [...] 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG [...] 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG [...] 05:31:50 DEBUG "value": [ 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG [...] 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG [...] 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG [...] 05:31:50 DEBUG 05:31:50 DEBUG [...] 05:31:50 DEBUG [...] 05:31:50 DEBUG [...] 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG [...] 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG "pod": "ds-cts-2" 05:31:50 DEBUG 05:31:50 DEBUG }, 05:31:50 DEBUG "pod": "ds-idrepo-2" 05:31:50 DEBUG [...] 05:31:50 DEBUG }, 05:31:50 DEBUG 05:31:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG 05:31:50 DEBUG "pod": "ds-cts-0" 05:31:50 DEBUG }, 05:31:50 DEBUG 05:31:50 DEBUG "value": [ 05:31:50 DEBUG }, 05:31:50 DEBUG 05:31:50 DEBUG "value": [ 05:31:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:31:50 DEBUG }, 05:31:50 DEBUG "instance": "10.142.0.43:9100" 05:31:50 DEBUG "pod": "ds-cts-2" 05:31:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:31:50 DEBUG }, 05:31:50 DEBUG "value": [ 05:31:50 DEBUG "pod": "ds-cts-0" 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG "value": [ 05:31:50 DEBUG 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG }, 05:31:50 DEBUG "value": [ 05:31:50 DEBUG }, 05:31:50 DEBUG }, 05:31:50 DEBUG }, 05:31:50 DEBUG "value": [ 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG }, 05:31:50 DEBUG "0" 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG 05:31:50 DEBUG "0.5804415817935958" 05:31:50 DEBUG "value": [ 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG "value": [ 05:31:50 DEBUG "value": [ 05:31:50 DEBUG "value": [ 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG "0" 05:31:50 DEBUG "value": [ 05:31:50 DEBUG ] 05:31:50 DEBUG "0.03333333333333333" 05:31:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:31:50 DEBUG ] 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG "0" 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG "18.11990592131857" 05:31:50 DEBUG ] 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG } 05:31:50 DEBUG ] 05:31:50 DEBUG }, 05:31:50 DEBUG } 05:31:50 DEBUG "0" 05:31:50 DEBUG ] 05:31:50 DEBUG "4910737.466666667" 05:31:50 DEBUG "0" 05:31:50 DEBUG "0" 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG "92597471.55558138" 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG "value": [ 05:31:50 DEBUG 1713760309, 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG ] 05:31:50 DEBUG ] 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG ] 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG ] 05:31:50 DEBUG ] 05:31:50 DEBUG "3507101696" 05:31:50 DEBUG } 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG ] 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG ] 05:31:50 DEBUG ] 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG ] 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:31:50 DEBUG } 05:32:06 INFO 05:32:06 INFO [loop_until]: kubectl --namespace=xlou top pods 05:32:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:32:07 INFO [loop_until]: OK (rc = 0) 05:32:07 DEBUG --- stdout --- 05:32:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 940m 8793Mi am-767f6b67b5-qcqz2 984m 8847Mi am-767f6b67b5-vbltz 977m 8785Mi ds-cts-0 17894m 12507Mi ds-cts-1 17344m 12544Mi ds-cts-2 18564m 12551Mi ds-idrepo-0 301m 3999Mi ds-idrepo-1 110m 4001Mi ds-idrepo-2 174m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1183Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 10m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1071m 3403Mi 05:32:07 DEBUG --- stderr --- 05:32:07 DEBUG 05:32:14 INFO 05:32:14 INFO [loop_until]: kubectl --namespace=xlou top node 05:32:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:32:14 INFO [loop_until]: OK (rc = 0) 05:32:14 DEBUG --- stdout --- 05:32:14 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6250Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 350m 1% 8058Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1020m 3% 11555Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17868m 59% 15770Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5208Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1068m 3% 11742Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3154Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1162m 3% 11032Mi 9% gke-xlou-cdm-ds-0f601751-j488 262m 0% 7456Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17379m 58% 17696Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 288m 0% 8032Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18429m 61% 15276Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1134m 7% 13343Mi 22% 05:32:14 DEBUG --- stderr --- 05:32:14 DEBUG 05:32:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760366" 05:32:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760366" 05:32:46 ERROR [http_cmd]: ERROR 05:32:46 ERROR --- status code --- 05:32:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:32:46 ERROR --- http response --- 05:32:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:32:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:32:46 INFO [http_cmd]: http status code OK 05:32:46 DEBUG --- status code --- 05:32:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:46 DEBUG --- http response --- 05:32:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760366" 05:32:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760366" 05:32:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760366" 05:32:46 INFO [http_cmd]: http status code OK 05:32:46 INFO [http_cmd]: http status code OK 05:32:46 DEBUG --- status code --- 05:32:46 DEBUG --- status code --- 05:32:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:46 DEBUG --- http response --- 05:32:46 DEBUG --- http response --- 05:32:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:32:46 INFO [http_cmd]: http status code OK 05:32:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:46 DEBUG ----- output ----- 05:32:46 DEBUG --- status code --- 05:32:46 DEBUG ----- output ----- 05:32:46 DEBUG { 05:32:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:46 DEBUG { 05:32:46 DEBUG "status": "success", 05:32:46 DEBUG --- http response --- 05:32:46 DEBUG "status": "success", 05:32:46 DEBUG "data": { 05:32:46 DEBUG "data": { 05:32:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:32:46 DEBUG "resultType": "vector", 05:32:46 DEBUG "resultType": "vector", 05:32:46 DEBUG "result": [ 05:32:46 DEBUG "result": [ 05:32:46 DEBUG { 05:32:46 DEBUG ----- output ----- 05:32:46 DEBUG { 05:32:46 DEBUG "metric": { 05:32:46 DEBUG { 05:32:46 DEBUG "metric": { 05:32:46 DEBUG "pod": "ds-idrepo-0" 05:32:46 DEBUG "status": "success", 05:32:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:32:46 DEBUG }, 05:32:46 DEBUG "data": { 05:32:46 DEBUG }, 05:32:46 DEBUG "value": [ 05:32:46 DEBUG "resultType": "vector", 05:32:46 DEBUG "value": [ 05:32:46 DEBUG 05:32:46 DEBUG "result": [ 05:32:46 DEBUG { 05:32:46 DEBUG 05:32:46 DEBUG 05:32:46 DEBUG "metric": { 05:32:46 DEBUG 05:32:46 DEBUG 05:32:46 DEBUG "pod": "ds-cts-0" 05:32:46 DEBUG [...] 05:32:46 DEBUG 05:32:46 DEBUG }, 05:32:46 DEBUG 05:32:46 DEBUG [...] 05:32:46 DEBUG "value": [ 05:32:46 DEBUG 05:32:46 DEBUG 05:32:46 DEBUG 05:32:46 DEBUG 05:32:46 DEBUG 05:32:46 DEBUG 05:32:46 DEBUG "pod": "ds-idrepo-2" 05:32:46 DEBUG 05:32:46 DEBUG 05:32:46 DEBUG }, 05:32:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:32:46 DEBUG [...] 05:32:46 DEBUG "value": [ 05:32:46 DEBUG }, 05:32:46 DEBUG 05:32:46 DEBUG 1713760366, 05:32:46 DEBUG "value": [ 05:32:46 DEBUG 05:32:46 DEBUG "0" 05:32:46 DEBUG 1713760366, 05:32:46 DEBUG 05:32:46 DEBUG ] 05:32:46 DEBUG "33198844.818886146" 05:32:46 DEBUG "pod": "ds-cts-2" 05:32:46 DEBUG } 05:32:46 DEBUG ] 05:32:46 DEBUG }, 05:32:46 DEBUG ] 05:32:46 DEBUG } 05:32:46 DEBUG "value": [ 05:32:46 DEBUG } 05:32:46 DEBUG ] 05:32:46 DEBUG 1713760366, 05:32:46 DEBUG } 05:32:46 DEBUG } 05:32:46 DEBUG "99.43173333333334" 05:32:46 DEBUG } 05:32:46 DEBUG ] 05:32:46 DEBUG } 05:32:46 DEBUG ] 05:32:46 DEBUG } 05:32:46 DEBUG } 05:32:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760367" 05:32:47 INFO [http_cmd]: http status code OK 05:32:47 DEBUG --- status code --- 05:32:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:47 DEBUG --- http response --- 05:32:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:47 DEBUG ----- output ----- 05:32:47 DEBUG { 05:32:47 DEBUG "status": "success", 05:32:47 DEBUG "data": { 05:32:47 DEBUG "resultType": "vector", 05:32:47 DEBUG "result": [ 05:32:47 DEBUG { 05:32:47 DEBUG "metric": { 05:32:47 DEBUG "instance": "10.142.0.107:9100" 05:32:47 DEBUG }, 05:32:47 DEBUG "value": [ 05:32:47 DEBUG 05:32:47 DEBUG 05:32:47 DEBUG 05:32:47 DEBUG [...] 05:32:47 DEBUG 05:32:47 DEBUG 05:32:47 DEBUG 05:32:47 DEBUG "instance": "10.142.0.43:9100" 05:32:47 DEBUG }, 05:32:47 DEBUG "value": [ 05:32:47 DEBUG 1713760367, 05:32:47 DEBUG "5803673996288" 05:32:47 DEBUG ] 05:32:47 DEBUG } 05:32:47 DEBUG ] 05:32:47 DEBUG } 05:32:47 DEBUG } 05:32:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760368" 05:32:48 INFO [http_cmd]: http status code OK 05:32:48 DEBUG --- status code --- 05:32:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:48 DEBUG --- http response --- 05:32:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:48 DEBUG ----- output ----- 05:32:48 DEBUG { 05:32:48 DEBUG "status": "success", 05:32:48 DEBUG "data": { 05:32:48 DEBUG "resultType": "vector", 05:32:48 DEBUG "result": [ 05:32:48 DEBUG { 05:32:48 DEBUG "metric": { 05:32:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:32:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760368" 05:32:48 DEBUG }, 05:32:48 DEBUG "value": [ 05:32:48 DEBUG 05:32:48 DEBUG 05:32:48 DEBUG 05:32:48 DEBUG [...] 05:32:48 DEBUG 05:32:48 DEBUG 05:32:48 DEBUG 05:32:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:32:48 DEBUG }, 05:32:48 DEBUG "value": [ 05:32:48 DEBUG 1713760368, 05:32:48 DEBUG "1000" 05:32:48 DEBUG ] 05:32:48 DEBUG } 05:32:48 DEBUG ] 05:32:48 DEBUG } 05:32:48 DEBUG } 05:32:48 INFO [http_cmd]: http status code OK 05:32:48 DEBUG --- status code --- 05:32:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:48 DEBUG --- http response --- 05:32:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:32:48 DEBUG ----- output ----- 05:32:48 DEBUG { 05:32:48 DEBUG "status": "success", 05:32:48 DEBUG "data": { 05:32:48 DEBUG "resultType": "vector", 05:32:48 DEBUG "result": [ 05:32:48 DEBUG { 05:32:48 DEBUG "metric": { 05:32:48 DEBUG "pod": "ds-cts-0" 05:32:48 DEBUG }, 05:32:48 DEBUG "value": [ 05:32:48 DEBUG 05:32:48 DEBUG 05:32:48 DEBUG 05:32:48 DEBUG [...] 05:32:48 DEBUG 05:32:48 DEBUG 05:32:48 DEBUG 05:32:48 DEBUG "pod": "ds-cts-2" 05:32:48 DEBUG }, 05:32:48 DEBUG "value": [ 05:32:48 DEBUG 1713760368, 05:32:48 DEBUG "36.53333333333333" 05:32:48 DEBUG ] 05:32:48 DEBUG } 05:32:48 DEBUG ] 05:32:48 DEBUG } 05:32:48 DEBUG } 05:32:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760369" 05:32:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760369" 05:32:49 INFO [http_cmd]: http status code OK 05:32:49 DEBUG --- status code --- 05:32:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:49 DEBUG --- http response --- 05:32:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:49 DEBUG ----- output ----- 05:32:49 DEBUG { 05:32:49 DEBUG "status": "success", 05:32:49 DEBUG "data": { 05:32:49 DEBUG "resultType": "vector", 05:32:49 DEBUG "result": [ 05:32:49 DEBUG { 05:32:49 DEBUG "metric": { 05:32:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG [...] 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 1713760369, 05:32:49 DEBUG "967.629" 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG } 05:32:49 INFO [http_cmd]: http status code OK 05:32:49 DEBUG --- status code --- 05:32:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:49 DEBUG --- http response --- 05:32:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:49 DEBUG ----- output ----- 05:32:49 DEBUG { 05:32:49 DEBUG "status": "success", 05:32:49 DEBUG "data": { 05:32:49 DEBUG "resultType": "vector", 05:32:49 DEBUG "result": [ 05:32:49 DEBUG { 05:32:49 DEBUG "metric": { 05:32:49 DEBUG "pod": "ds-idrepo-0" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG [...] 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG "pod": "ds-idrepo-2" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 1713760369, 05:32:49 DEBUG "0" 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG } 05:32:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760369" 05:32:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760369" 05:32:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760369" 05:32:49 INFO [http_cmd]: http status code OK 05:32:49 DEBUG --- status code --- 05:32:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:49 DEBUG --- http response --- 05:32:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:32:49 DEBUG ----- output ----- 05:32:49 DEBUG { 05:32:49 DEBUG "status": "success", 05:32:49 DEBUG "data": { 05:32:49 DEBUG "resultType": "vector", 05:32:49 DEBUG "result": [ 05:32:49 DEBUG { 05:32:49 DEBUG "metric": { 05:32:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG [...] 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 1713760369, 05:32:49 DEBUG "0" 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG } 05:32:49 INFO [http_cmd]: http status code OK 05:32:49 DEBUG --- status code --- 05:32:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:49 DEBUG --- http response --- 05:32:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:49 DEBUG ----- output ----- 05:32:49 DEBUG { 05:32:49 DEBUG "status": "success", 05:32:49 DEBUG "data": { 05:32:49 DEBUG "resultType": "vector", 05:32:49 DEBUG "result": [ 05:32:49 DEBUG { 05:32:49 DEBUG "metric": { 05:32:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG [...] 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 1713760369, 05:32:49 DEBUG "2" 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG } 05:32:49 INFO [http_cmd]: http status code OK 05:32:49 DEBUG --- status code --- 05:32:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:49 DEBUG --- http response --- 05:32:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:49 DEBUG ----- output ----- 05:32:49 DEBUG { 05:32:49 DEBUG "status": "success", 05:32:49 DEBUG "data": { 05:32:49 DEBUG "resultType": "vector", 05:32:49 DEBUG "result": [ 05:32:49 DEBUG { 05:32:49 DEBUG "metric": { 05:32:49 DEBUG "instance": "10.142.0.107:9100" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG [...] 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG "instance": "10.142.0.43:9100" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 1713760369, 05:32:49 DEBUG "0.47899999999995085" 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG } 05:32:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760369" 05:32:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760369" 05:32:49 INFO [http_cmd]: http status code OK 05:32:49 DEBUG --- status code --- 05:32:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:49 DEBUG --- http response --- 05:32:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:49 DEBUG ----- output ----- 05:32:49 DEBUG { 05:32:49 DEBUG "status": "success", 05:32:49 DEBUG "data": { 05:32:49 DEBUG "resultType": "vector", 05:32:49 DEBUG "result": [ 05:32:49 DEBUG { 05:32:49 DEBUG "metric": { 05:32:49 DEBUG "instance": "10.142.0.107:9100" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG [...] 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG "instance": "10.142.0.43:9100" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 1713760369, 05:32:49 DEBUG "4993016.9" 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG } 05:32:49 INFO [http_cmd]: http status code OK 05:32:49 DEBUG --- status code --- 05:32:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:49 DEBUG --- http response --- 05:32:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:49 DEBUG ----- output ----- 05:32:49 DEBUG { 05:32:49 DEBUG "status": "success", 05:32:49 DEBUG "data": { 05:32:49 DEBUG "resultType": "vector", 05:32:49 DEBUG "result": [ 05:32:49 DEBUG { 05:32:49 DEBUG "metric": { 05:32:49 DEBUG "instance": "10.142.0.107:9100" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG [...] 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG 05:32:49 DEBUG "instance": "10.142.0.43:9100" 05:32:49 DEBUG }, 05:32:49 DEBUG "value": [ 05:32:49 DEBUG 1713760369, 05:32:49 DEBUG "0.06099999999999759" 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG ] 05:32:49 DEBUG } 05:32:49 DEBUG } 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760370" 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG { 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG "data": { 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG "result": [ 05:32:50 DEBUG { 05:32:50 DEBUG "metric": { 05:32:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:32:50 DEBUG }, 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG [...] 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:32:50 DEBUG }, 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG "3832680448" 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760370" 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760370" 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760370" 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760370" 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760370" 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760370" 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760370" 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760370" 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760370" 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760370" 05:32:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760370" 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG --- status code --- 05:32:50 INFO [http_cmd]: http status code OK 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG --- status code --- 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:50 DEBUG { 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG { 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:32:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG { 05:32:50 DEBUG { 05:32:50 DEBUG { 05:32:50 DEBUG { 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG --- http response --- 05:32:50 DEBUG { 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG { 05:32:50 DEBUG "data": { 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG "data": { 05:32:50 DEBUG { 05:32:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG "data": { 05:32:50 DEBUG "data": { 05:32:50 DEBUG "data": { 05:32:50 DEBUG "data": { 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG { 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG ----- output ----- 05:32:50 DEBUG "data": { 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG "data": { 05:32:50 DEBUG "result": [ 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG "result": [ 05:32:50 DEBUG "data": { 05:32:50 DEBUG { 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG "result": [ 05:32:50 DEBUG "result": [ 05:32:50 DEBUG "result": [ 05:32:50 DEBUG "result": [ 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG { 05:32:50 DEBUG "data": { 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG "status": "success", 05:32:50 DEBUG "result": [ 05:32:50 DEBUG { 05:32:50 DEBUG { 05:32:50 DEBUG { 05:32:50 DEBUG { 05:32:50 DEBUG "result": [ 05:32:50 DEBUG "metric": { 05:32:50 DEBUG "result": [ 05:32:50 DEBUG { 05:32:50 DEBUG "result": [ 05:32:50 DEBUG "data": { 05:32:50 DEBUG { 05:32:50 DEBUG "metric": { 05:32:50 DEBUG "metric": { 05:32:50 DEBUG "metric": { 05:32:50 DEBUG "metric": { 05:32:50 DEBUG { 05:32:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:32:50 DEBUG { 05:32:50 DEBUG "metric": { 05:32:50 DEBUG { 05:32:50 DEBUG "resultType": "vector", 05:32:50 DEBUG "metric": { 05:32:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:32:50 DEBUG "pod": "ds-idrepo-0" 05:32:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:32:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:32:50 DEBUG "metric": { 05:32:50 DEBUG }, 05:32:50 DEBUG "metric": { 05:32:50 DEBUG "pod": "ds-idrepo-0" 05:32:50 DEBUG "metric": { 05:32:50 DEBUG "result": [ 05:32:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:32:50 DEBUG }, 05:32:50 DEBUG "value": [ 05:32:50 DEBUG }, 05:32:50 DEBUG "value": [ 05:32:50 DEBUG "instance": "10.142.0.107:9100" 05:32:50 DEBUG "value": [ 05:32:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:32:50 DEBUG }, 05:32:50 DEBUG "value": [ 05:32:50 DEBUG { 05:32:50 DEBUG }, 05:32:50 DEBUG }, 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG }, 05:32:50 DEBUG 05:32:50 DEBUG }, 05:32:50 DEBUG }, 05:32:50 DEBUG 05:32:50 DEBUG "pod": "ds-cts-0" 05:32:50 DEBUG "metric": { 05:32:50 DEBUG "value": [ 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 05:32:50 DEBUG "value": [ 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 05:32:50 DEBUG }, 05:32:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG "value": [ 05:32:50 DEBUG }, 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG [...] 05:32:50 DEBUG [...] 05:32:50 DEBUG 05:32:50 DEBUG [...] 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG [...] 05:32:50 DEBUG 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG [...] 05:32:50 DEBUG [...] 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG [...] 05:32:50 DEBUG 05:32:50 DEBUG [...] 05:32:50 DEBUG [...] 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG [...] 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:32:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:32:50 DEBUG 05:32:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:32:50 DEBUG }, 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG [...] 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG }, 05:32:50 DEBUG }, 05:32:50 DEBUG 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 05:32:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:32:50 DEBUG "pod": "ds-idrepo-2" 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG "pod": "ds-cts-0" 05:32:50 DEBUG "pod": "ds-cts-2" 05:32:50 DEBUG }, 05:32:50 DEBUG "value": [ 05:32:50 DEBUG "instance": "10.142.0.43:9100" 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG 05:32:50 DEBUG }, 05:32:50 DEBUG }, 05:32:50 DEBUG 05:32:50 DEBUG 05:32:50 DEBUG }, 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG }, 05:32:50 DEBUG "0" 05:32:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG "pod": "ds-cts-2" 05:32:50 DEBUG 05:32:50 DEBUG "value": [ 05:32:50 DEBUG "0" 05:32:50 DEBUG "value": [ 05:32:50 DEBUG "1.0319583851377698" 05:32:50 DEBUG "value": [ 05:32:50 DEBUG ] 05:32:50 DEBUG }, 05:32:50 DEBUG "0" 05:32:50 DEBUG }, 05:32:50 DEBUG "value": [ 05:32:50 DEBUG "pod": "ds-cts-0" 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG ] 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG ] 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG } 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG "value": [ 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG }, 05:32:50 DEBUG "94272952.00673589" 05:32:50 DEBUG } 05:32:50 DEBUG "0" 05:32:50 DEBUG } 05:32:50 DEBUG "4990863.3" 05:32:50 DEBUG ] 05:32:50 DEBUG "0" 05:32:50 DEBUG ] 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG "0" 05:32:50 DEBUG "value": [ 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG ] 05:32:50 DEBUG ] 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG "0" 05:32:50 DEBUG ] 05:32:50 DEBUG 1713760370, 05:32:50 DEBUG ] 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG ] 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG "18.146766488919127" 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG ] 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG ] 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG ] 05:32:50 DEBUG } 05:32:50 DEBUG } 05:32:50 DEBUG } 05:33:07 INFO 05:33:07 INFO [loop_until]: kubectl --namespace=xlou top pods 05:33:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:07 INFO [loop_until]: OK (rc = 0) 05:33:07 DEBUG --- stdout --- 05:33:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 982m 8792Mi am-767f6b67b5-qcqz2 970m 8841Mi am-767f6b67b5-vbltz 951m 8780Mi ds-cts-0 17745m 12507Mi ds-cts-1 17279m 12546Mi ds-cts-2 18442m 12551Mi ds-idrepo-0 14m 3999Mi ds-idrepo-1 12m 4001Mi ds-idrepo-2 12m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1183Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1015m 3719Mi 05:33:07 DEBUG --- stderr --- 05:33:07 DEBUG 05:33:14 INFO 05:33:14 INFO [loop_until]: kubectl --namespace=xlou top node 05:33:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:14 INFO [loop_until]: OK (rc = 0) 05:33:14 DEBUG --- stdout --- 05:33:14 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 6246Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 8077Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1035m 3% 11533Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17136m 57% 15763Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5185Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1047m 3% 11727Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3148Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1131m 3% 11014Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 7473Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17276m 57% 17722Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 91m 0% 8045Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18412m 61% 15268Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 13674Mi 23% 05:33:14 DEBUG --- stderr --- 05:33:14 DEBUG 05:33:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760426" 05:33:46 ERROR [http_cmd]: ERROR 05:33:46 ERROR --- status code --- 05:33:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:33:46 ERROR --- http response --- 05:33:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:33:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:33:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760426" 05:33:47 INFO [http_cmd]: http status code OK 05:33:47 DEBUG --- status code --- 05:33:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:47 DEBUG --- http response --- 05:33:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760427" 05:33:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760427" 05:33:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760427" 05:33:47 INFO [http_cmd]: http status code OK 05:33:47 DEBUG --- status code --- 05:33:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:47 DEBUG --- http response --- 05:33:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:47 DEBUG ----- output ----- 05:33:47 DEBUG { 05:33:47 DEBUG "status": "success", 05:33:47 DEBUG "data": { 05:33:47 DEBUG "resultType": "vector", 05:33:47 DEBUG "result": [ 05:33:47 DEBUG { 05:33:47 DEBUG "metric": { 05:33:47 DEBUG "pod": "ds-idrepo-0" 05:33:47 DEBUG }, 05:33:47 DEBUG "value": [ 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG [...] 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG "pod": "ds-idrepo-2" 05:33:47 DEBUG }, 05:33:47 DEBUG "value": [ 05:33:47 DEBUG 1713760427, 05:33:47 DEBUG "0" 05:33:47 DEBUG ] 05:33:47 DEBUG } 05:33:47 DEBUG ] 05:33:47 DEBUG } 05:33:47 DEBUG } 05:33:47 INFO [http_cmd]: http status code OK 05:33:47 DEBUG --- status code --- 05:33:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:47 DEBUG --- http response --- 05:33:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:47 DEBUG ----- output ----- 05:33:47 DEBUG { 05:33:47 DEBUG "status": "success", 05:33:47 DEBUG "data": { 05:33:47 DEBUG "resultType": "vector", 05:33:47 DEBUG "result": [ 05:33:47 DEBUG { 05:33:47 DEBUG "metric": { 05:33:47 DEBUG "pod": "ds-cts-0" 05:33:47 DEBUG }, 05:33:47 DEBUG "value": [ 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG [...] 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG "pod": "ds-cts-2" 05:33:47 DEBUG }, 05:33:47 DEBUG "value": [ 05:33:47 DEBUG 1713760427, 05:33:47 DEBUG "97.46046666666666" 05:33:47 DEBUG ] 05:33:47 DEBUG } 05:33:47 DEBUG ] 05:33:47 DEBUG } 05:33:47 DEBUG } 05:33:47 INFO [http_cmd]: http status code OK 05:33:47 DEBUG --- status code --- 05:33:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:47 DEBUG --- http response --- 05:33:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:47 DEBUG ----- output ----- 05:33:47 DEBUG { 05:33:47 DEBUG "status": "success", 05:33:47 DEBUG "data": { 05:33:47 DEBUG "resultType": "vector", 05:33:47 DEBUG "result": [ 05:33:47 DEBUG { 05:33:47 DEBUG "metric": { 05:33:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:33:47 DEBUG }, 05:33:47 DEBUG "value": [ 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG [...] 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG 05:33:47 DEBUG "pod": "ds-cts-0" 05:33:47 DEBUG }, 05:33:47 DEBUG "value": [ 05:33:47 DEBUG 1713760427, 05:33:47 DEBUG "489629456.1468985" 05:33:47 DEBUG ] 05:33:47 DEBUG } 05:33:47 DEBUG ] 05:33:47 DEBUG } 05:33:47 DEBUG } 05:33:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760428" 05:33:48 INFO [http_cmd]: http status code OK 05:33:48 DEBUG --- status code --- 05:33:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:48 DEBUG --- http response --- 05:33:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:48 DEBUG ----- output ----- 05:33:48 DEBUG { 05:33:48 DEBUG "status": "success", 05:33:48 DEBUG "data": { 05:33:48 DEBUG "resultType": "vector", 05:33:48 DEBUG "result": [ 05:33:48 DEBUG { 05:33:48 DEBUG "metric": { 05:33:48 DEBUG "instance": "10.142.0.107:9100" 05:33:48 DEBUG }, 05:33:48 DEBUG "value": [ 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG [...] 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG "instance": "10.142.0.43:9100" 05:33:48 DEBUG }, 05:33:48 DEBUG "value": [ 05:33:48 DEBUG 1713760428, 05:33:48 DEBUG "5807557590016" 05:33:48 DEBUG ] 05:33:48 DEBUG } 05:33:48 DEBUG ] 05:33:48 DEBUG } 05:33:48 DEBUG } 05:33:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760428" 05:33:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760428" 05:33:48 INFO [http_cmd]: http status code OK 05:33:48 DEBUG --- status code --- 05:33:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:48 DEBUG --- http response --- 05:33:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:48 DEBUG ----- output ----- 05:33:48 DEBUG { 05:33:48 DEBUG "status": "success", 05:33:48 DEBUG "data": { 05:33:48 DEBUG "resultType": "vector", 05:33:48 DEBUG "result": [ 05:33:48 DEBUG { 05:33:48 DEBUG "metric": { 05:33:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:48 DEBUG }, 05:33:48 DEBUG "value": [ 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG [...] 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:48 DEBUG }, 05:33:48 DEBUG "value": [ 05:33:48 DEBUG 1713760428, 05:33:48 DEBUG "1000" 05:33:48 DEBUG ] 05:33:48 DEBUG } 05:33:48 DEBUG ] 05:33:48 DEBUG } 05:33:48 DEBUG } 05:33:48 INFO [http_cmd]: http status code OK 05:33:48 DEBUG --- status code --- 05:33:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:48 DEBUG --- http response --- 05:33:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:48 DEBUG ----- output ----- 05:33:48 DEBUG { 05:33:48 DEBUG "status": "success", 05:33:48 DEBUG "data": { 05:33:48 DEBUG "resultType": "vector", 05:33:48 DEBUG "result": [ 05:33:48 DEBUG { 05:33:48 DEBUG "metric": { 05:33:48 DEBUG "pod": "ds-cts-0" 05:33:48 DEBUG }, 05:33:48 DEBUG "value": [ 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG [...] 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG 05:33:48 DEBUG "pod": "ds-cts-2" 05:33:48 DEBUG }, 05:33:48 DEBUG "value": [ 05:33:48 DEBUG 1713760428, 05:33:48 DEBUG "38.43333333333333" 05:33:48 DEBUG ] 05:33:48 DEBUG } 05:33:48 DEBUG ] 05:33:48 DEBUG } 05:33:48 DEBUG } 05:33:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760429" 05:33:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760429" 05:33:49 INFO [http_cmd]: http status code OK 05:33:49 DEBUG --- status code --- 05:33:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:49 DEBUG --- http response --- 05:33:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:49 DEBUG ----- output ----- 05:33:49 DEBUG { 05:33:49 DEBUG "status": "success", 05:33:49 DEBUG "data": { 05:33:49 DEBUG "resultType": "vector", 05:33:49 DEBUG "result": [ 05:33:49 DEBUG { 05:33:49 DEBUG "metric": { 05:33:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG [...] 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 1713760429, 05:33:49 DEBUG "970.032" 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG } 05:33:49 INFO [http_cmd]: http status code OK 05:33:49 DEBUG --- status code --- 05:33:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:49 DEBUG --- http response --- 05:33:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:49 DEBUG ----- output ----- 05:33:49 DEBUG { 05:33:49 DEBUG "status": "success", 05:33:49 DEBUG "data": { 05:33:49 DEBUG "resultType": "vector", 05:33:49 DEBUG "result": [ 05:33:49 DEBUG { 05:33:49 DEBUG "metric": { 05:33:49 DEBUG "pod": "ds-idrepo-0" 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG [...] 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG "pod": "ds-idrepo-2" 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 1713760429, 05:33:49 DEBUG "0" 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG } 05:33:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760429" 05:33:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760429" 05:33:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760429" 05:33:49 INFO [http_cmd]: http status code OK 05:33:49 INFO [http_cmd]: http status code OK 05:33:49 DEBUG --- status code --- 05:33:49 DEBUG --- status code --- 05:33:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:49 DEBUG --- http response --- 05:33:49 DEBUG --- http response --- 05:33:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:33:49 DEBUG ----- output ----- 05:33:49 DEBUG ----- output ----- 05:33:49 DEBUG { 05:33:49 DEBUG { 05:33:49 DEBUG "status": "success", 05:33:49 DEBUG "status": "success", 05:33:49 DEBUG "data": { 05:33:49 DEBUG "data": { 05:33:49 DEBUG "resultType": "vector", 05:33:49 DEBUG "resultType": "vector", 05:33:49 DEBUG "result": [ 05:33:49 DEBUG "result": [ 05:33:49 DEBUG { 05:33:49 DEBUG { 05:33:49 DEBUG "metric": { 05:33:49 DEBUG "metric": { 05:33:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:33:49 DEBUG }, 05:33:49 INFO [http_cmd]: http status code OK 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG --- status code --- 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 05:33:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG --- http response --- 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:49 DEBUG 05:33:49 DEBUG [...] 05:33:49 DEBUG ----- output ----- 05:33:49 DEBUG [...] 05:33:49 DEBUG 05:33:49 DEBUG { 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG "status": "success", 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG "data": { 05:33:49 DEBUG 05:33:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:49 DEBUG "resultType": "vector", 05:33:49 DEBUG "pod": "ds-cts-0" 05:33:49 DEBUG }, 05:33:49 DEBUG "result": [ 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG { 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 1713760429, 05:33:49 DEBUG "metric": { 05:33:49 DEBUG 1713760429, 05:33:49 DEBUG "2" 05:33:49 DEBUG "instance": "10.142.0.107:9100" 05:33:49 DEBUG }, 05:33:49 DEBUG ] 05:33:49 DEBUG "71351482.27215047" 05:33:49 DEBUG "value": [ 05:33:49 DEBUG } 05:33:49 DEBUG ] 05:33:49 DEBUG 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG 05:33:49 DEBUG } 05:33:49 DEBUG ] 05:33:49 DEBUG 05:33:49 DEBUG } 05:33:49 DEBUG } 05:33:49 DEBUG [...] 05:33:49 DEBUG } 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG "instance": "10.142.0.43:9100" 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 1713760429, 05:33:49 DEBUG "0.4754666666666253" 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG } 05:33:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760429" 05:33:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760429" 05:33:49 INFO [http_cmd]: http status code OK 05:33:49 DEBUG --- status code --- 05:33:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:49 DEBUG --- http response --- 05:33:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:49 DEBUG ----- output ----- 05:33:49 DEBUG { 05:33:49 DEBUG "status": "success", 05:33:49 DEBUG "data": { 05:33:49 DEBUG "resultType": "vector", 05:33:49 DEBUG "result": [ 05:33:49 DEBUG { 05:33:49 DEBUG "metric": { 05:33:49 DEBUG "instance": "10.142.0.107:9100" 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG [...] 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG "instance": "10.142.0.43:9100" 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 1713760429, 05:33:49 DEBUG "4934085.6" 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG } 05:33:49 INFO [http_cmd]: http status code OK 05:33:49 DEBUG --- status code --- 05:33:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:49 DEBUG --- http response --- 05:33:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:49 DEBUG ----- output ----- 05:33:49 DEBUG { 05:33:49 DEBUG "status": "success", 05:33:49 DEBUG "data": { 05:33:49 DEBUG "resultType": "vector", 05:33:49 DEBUG "result": [ 05:33:49 DEBUG { 05:33:49 DEBUG "metric": { 05:33:49 DEBUG "instance": "10.142.0.107:9100" 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG [...] 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG 05:33:49 DEBUG "instance": "10.142.0.43:9100" 05:33:49 DEBUG }, 05:33:49 DEBUG "value": [ 05:33:49 DEBUG 1713760429, 05:33:49 DEBUG "0.06099999999999758" 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG ] 05:33:49 DEBUG } 05:33:49 DEBUG } 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760430" 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 DEBUG --- status code --- 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG { 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG "data": { 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG "result": [ 05:33:50 DEBUG { 05:33:50 DEBUG "metric": { 05:33:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:33:50 DEBUG }, 05:33:50 DEBUG "value": [ 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG [...] 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:33:50 DEBUG }, 05:33:50 DEBUG "value": [ 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG "4173885440" 05:33:50 DEBUG ] 05:33:50 DEBUG } 05:33:50 DEBUG ] 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760430" 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760430" 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760430" 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760430" 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760430" 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760430" 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760430" 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760430" 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760430" 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760430" 05:33:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760430" 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 DEBUG --- status code --- 05:33:50 DEBUG --- status code --- 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG --- status code --- 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 DEBUG --- status code --- 05:33:50 DEBUG --- status code --- 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 INFO [http_cmd]: http status code OK 05:33:50 DEBUG --- status code --- 05:33:50 DEBUG --- status code --- 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG --- status code --- 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG --- status code --- 05:33:50 DEBUG --- status code --- 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG --- status code --- 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG { 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG --- http response --- 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:50 DEBUG "data": { 05:33:50 DEBUG { 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:50 DEBUG { 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:50 DEBUG { 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:50 DEBUG { 05:33:50 DEBUG { 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG { 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG ----- output ----- 05:33:50 DEBUG "data": { 05:33:50 DEBUG "result": [ 05:33:50 DEBUG "data": { 05:33:50 DEBUG { 05:33:50 DEBUG "data": { 05:33:50 DEBUG { 05:33:50 DEBUG "data": { 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG { 05:33:50 DEBUG "data": { 05:33:50 DEBUG { 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG { 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG "data": { 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG "status": "success", 05:33:50 DEBUG "result": [ 05:33:50 DEBUG "metric": { 05:33:50 DEBUG "result": [ 05:33:50 DEBUG "data": { 05:33:50 DEBUG "result": [ 05:33:50 DEBUG "data": { 05:33:50 DEBUG "result": [ 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG "data": { 05:33:50 DEBUG "result": [ 05:33:50 DEBUG { 05:33:50 DEBUG { 05:33:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:33:50 DEBUG { 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG { 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG { 05:33:50 DEBUG "result": [ 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG "data": { 05:33:50 DEBUG "metric": { 05:33:50 DEBUG "metric": { 05:33:50 DEBUG }, 05:33:50 DEBUG "metric": { 05:33:50 DEBUG "result": [ 05:33:50 DEBUG "metric": { 05:33:50 DEBUG "result": [ 05:33:50 DEBUG "metric": { 05:33:50 DEBUG { 05:33:50 DEBUG "result": [ 05:33:50 DEBUG "resultType": "vector", 05:33:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:50 DEBUG "instance": "10.142.0.107:9100" 05:33:50 DEBUG "value": [ 05:33:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:50 DEBUG { 05:33:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:50 DEBUG { 05:33:50 DEBUG "pod": "ds-idrepo-0" 05:33:50 DEBUG "metric": { 05:33:50 DEBUG { 05:33:50 DEBUG "result": [ 05:33:50 DEBUG }, 05:33:50 DEBUG }, 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG "metric": { 05:33:50 DEBUG }, 05:33:50 DEBUG "metric": { 05:33:50 DEBUG }, 05:33:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:33:50 DEBUG "metric": { 05:33:50 DEBUG { 05:33:50 DEBUG "value": [ 05:33:50 DEBUG "value": [ 05:33:50 DEBUG 05:33:50 DEBUG "value": [ 05:33:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:50 DEBUG "value": [ 05:33:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:33:50 DEBUG "value": [ 05:33:50 DEBUG }, 05:33:50 DEBUG "value": [ 05:33:50 DEBUG "metric": { 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG 05:33:50 DEBUG "pod": "ds-cts-0" 05:33:50 DEBUG 05:33:50 DEBUG "pod": "ds-idrepo-0" 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG [...] 05:33:50 DEBUG 05:33:50 DEBUG "value": [ 05:33:50 DEBUG 05:33:50 DEBUG "value": [ 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG "value": [ 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG [...] 05:33:50 DEBUG [...] 05:33:50 DEBUG 05:33:50 DEBUG [...] 05:33:50 DEBUG 05:33:50 DEBUG [...] 05:33:50 DEBUG 05:33:50 DEBUG [...] 05:33:50 DEBUG "value": [ 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG [...] 05:33:50 DEBUG 05:33:50 DEBUG [...] 05:33:50 DEBUG 05:33:50 DEBUG "pod": "ds-cts-0" 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG [...] 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:50 DEBUG [...] 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG 05:33:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:50 DEBUG 05:33:50 DEBUG "instance": "10.142.0.43:9100" 05:33:50 DEBUG "value": [ 05:33:50 DEBUG }, 05:33:50 DEBUG 05:33:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:50 DEBUG 05:33:50 DEBUG "pod": "ds-cts-2" 05:33:50 DEBUG [...] 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG "99389683.06337239" 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG 05:33:50 DEBUG "pod": "ds-cts-2" 05:33:50 DEBUG "value": [ 05:33:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:33:50 DEBUG "value": [ 05:33:50 DEBUG "value": [ 05:33:50 DEBUG ] 05:33:50 DEBUG 05:33:50 DEBUG "value": [ 05:33:50 DEBUG "pod": "ds-cts-0" 05:33:50 DEBUG "value": [ 05:33:50 DEBUG 05:33:50 DEBUG }, 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG }, 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG } 05:33:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG "0" 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG 05:33:50 DEBUG "value": [ 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG "value": [ 05:33:50 DEBUG "4928209.966666667" 05:33:50 DEBUG "0" 05:33:50 DEBUG ] 05:33:50 DEBUG }, 05:33:50 DEBUG ] 05:33:50 DEBUG }, 05:33:50 DEBUG "0" 05:33:50 DEBUG "pod": "ds-idrepo-2" 05:33:50 DEBUG "0" 05:33:50 DEBUG "0" 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG ] 05:33:50 DEBUG ] 05:33:50 DEBUG } 05:33:50 DEBUG "value": [ 05:33:50 DEBUG } 05:33:50 DEBUG "value": [ 05:33:50 DEBUG ] 05:33:50 DEBUG }, 05:33:50 DEBUG ] 05:33:50 DEBUG ] 05:33:50 DEBUG "1.0358524985439288" 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG ] 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG } 05:33:50 DEBUG "value": [ 05:33:50 DEBUG 1713760430, 05:33:50 DEBUG } 05:33:50 DEBUG ] 05:33:50 DEBUG ] 05:33:50 DEBUG ] 05:33:50 DEBUG "0" 05:33:50 DEBUG ] 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG "0" 05:33:50 DEBUG } 05:33:50 DEBUG ] 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG "17.90297712174572" 05:33:50 DEBUG } 05:33:50 DEBUG ] 05:33:50 DEBUG ] 05:33:50 DEBUG ] 05:33:50 DEBUG } 05:33:50 DEBUG ] 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG ] 05:33:50 DEBUG ] 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG ] 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG ] 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:33:50 DEBUG } 05:34:07 INFO 05:34:07 INFO [loop_until]: kubectl --namespace=xlou top pods 05:34:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:34:07 INFO [loop_until]: OK (rc = 0) 05:34:07 DEBUG --- stdout --- 05:34:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 953m 8791Mi am-767f6b67b5-qcqz2 987m 8840Mi am-767f6b67b5-vbltz 962m 8780Mi ds-cts-0 18349m 12509Mi ds-cts-1 17585m 12545Mi ds-cts-2 18334m 12549Mi ds-idrepo-0 14m 4000Mi ds-idrepo-1 13m 4001Mi ds-idrepo-2 11m 3985Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 6m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 998m 4053Mi 05:34:07 DEBUG --- stderr --- 05:34:07 DEBUG 05:34:14 INFO 05:34:14 INFO [loop_until]: kubectl --namespace=xlou top node 05:34:14 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:34:14 INFO [loop_until]: OK (rc = 0) 05:34:14 DEBUG --- stdout --- 05:34:14 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6243Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 8085Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1029m 3% 11526Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3348Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 17896m 59% 15760Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5199Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1043m 3% 11743Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3142Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1113m 3% 11019Mi 9% gke-xlou-cdm-ds-0f601751-j488 67m 0% 7475Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17168m 57% 17721Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 90m 0% 8062Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18568m 62% 15267Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 14010Mi 23% 05:34:14 DEBUG --- stderr --- 05:34:14 DEBUG 05:34:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760487" 05:34:47 ERROR [http_cmd]: ERROR 05:34:47 ERROR --- status code --- 05:34:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:34:47 ERROR --- http response --- 05:34:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:34:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:34:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760487" 05:34:47 INFO [http_cmd]: http status code OK 05:34:47 DEBUG --- status code --- 05:34:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:47 DEBUG --- http response --- 05:34:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760487" 05:34:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760487" 05:34:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760487" 05:34:47 INFO [http_cmd]: http status code OK 05:34:47 DEBUG --- status code --- 05:34:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:47 DEBUG --- http response --- 05:34:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:47 DEBUG ----- output ----- 05:34:47 DEBUG { 05:34:47 DEBUG "status": "success", 05:34:47 DEBUG "data": { 05:34:47 DEBUG "resultType": "vector", 05:34:47 DEBUG "result": [ 05:34:47 DEBUG { 05:34:47 DEBUG "metric": { 05:34:47 DEBUG "pod": "ds-idrepo-0" 05:34:47 DEBUG }, 05:34:47 DEBUG "value": [ 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG [...] 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG "pod": "ds-idrepo-2" 05:34:47 DEBUG }, 05:34:47 DEBUG "value": [ 05:34:47 DEBUG 1713760487, 05:34:47 DEBUG "0" 05:34:47 DEBUG ] 05:34:47 DEBUG } 05:34:47 DEBUG ] 05:34:47 DEBUG } 05:34:47 DEBUG } 05:34:47 INFO [http_cmd]: http status code OK 05:34:47 DEBUG --- status code --- 05:34:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:47 DEBUG --- http response --- 05:34:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:47 DEBUG ----- output ----- 05:34:47 DEBUG { 05:34:47 DEBUG "status": "success", 05:34:47 DEBUG "data": { 05:34:47 DEBUG "resultType": "vector", 05:34:47 DEBUG "result": [ 05:34:47 DEBUG { 05:34:47 DEBUG "metric": { 05:34:47 DEBUG "pod": "ds-cts-0" 05:34:47 DEBUG }, 05:34:47 DEBUG "value": [ 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG [...] 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG "pod": "ds-cts-2" 05:34:47 DEBUG }, 05:34:47 DEBUG "value": [ 05:34:47 DEBUG 1713760487, 05:34:47 DEBUG "95.29063333333332" 05:34:47 DEBUG ] 05:34:47 DEBUG } 05:34:47 DEBUG ] 05:34:47 DEBUG } 05:34:47 DEBUG } 05:34:47 INFO [http_cmd]: http status code OK 05:34:47 DEBUG --- status code --- 05:34:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:47 DEBUG --- http response --- 05:34:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:47 DEBUG ----- output ----- 05:34:47 DEBUG { 05:34:47 DEBUG "status": "success", 05:34:47 DEBUG "data": { 05:34:47 DEBUG "resultType": "vector", 05:34:47 DEBUG "result": [ 05:34:47 DEBUG { 05:34:47 DEBUG "metric": { 05:34:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:34:47 DEBUG }, 05:34:47 DEBUG "value": [ 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG [...] 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG 05:34:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:47 DEBUG }, 05:34:47 DEBUG "value": [ 05:34:47 DEBUG 1713760487, 05:34:47 DEBUG "0" 05:34:47 DEBUG ] 05:34:47 DEBUG } 05:34:47 DEBUG ] 05:34:47 DEBUG } 05:34:47 DEBUG } 05:34:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760488" 05:34:48 INFO [http_cmd]: http status code OK 05:34:48 DEBUG --- status code --- 05:34:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:48 DEBUG --- http response --- 05:34:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:48 DEBUG ----- output ----- 05:34:48 DEBUG { 05:34:48 DEBUG "status": "success", 05:34:48 DEBUG "data": { 05:34:48 DEBUG "resultType": "vector", 05:34:48 DEBUG "result": [ 05:34:48 DEBUG { 05:34:48 DEBUG "metric": { 05:34:48 DEBUG "instance": "10.142.0.107:9100" 05:34:48 DEBUG }, 05:34:48 DEBUG "value": [ 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG [...] 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG "instance": "10.142.0.43:9100" 05:34:48 DEBUG }, 05:34:48 DEBUG "value": [ 05:34:48 DEBUG 1713760488, 05:34:48 DEBUG "5811467099136" 05:34:48 DEBUG ] 05:34:48 DEBUG } 05:34:48 DEBUG ] 05:34:48 DEBUG } 05:34:48 DEBUG } 05:34:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760488" 05:34:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760488" 05:34:48 INFO [http_cmd]: http status code OK 05:34:48 DEBUG --- status code --- 05:34:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:48 DEBUG --- http response --- 05:34:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:48 DEBUG ----- output ----- 05:34:48 DEBUG { 05:34:48 DEBUG "status": "success", 05:34:48 DEBUG "data": { 05:34:48 DEBUG "resultType": "vector", 05:34:48 DEBUG "result": [ 05:34:48 DEBUG { 05:34:48 DEBUG "metric": { 05:34:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:48 DEBUG }, 05:34:48 DEBUG "value": [ 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG [...] 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:48 DEBUG }, 05:34:48 DEBUG "value": [ 05:34:48 DEBUG 1713760488, 05:34:48 DEBUG "1000" 05:34:48 DEBUG ] 05:34:48 DEBUG } 05:34:48 DEBUG ] 05:34:48 DEBUG } 05:34:48 DEBUG } 05:34:48 INFO [http_cmd]: http status code OK 05:34:48 DEBUG --- status code --- 05:34:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:48 DEBUG --- http response --- 05:34:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:48 DEBUG ----- output ----- 05:34:48 DEBUG { 05:34:48 DEBUG "status": "success", 05:34:48 DEBUG "data": { 05:34:48 DEBUG "resultType": "vector", 05:34:48 DEBUG "result": [ 05:34:48 DEBUG { 05:34:48 DEBUG "metric": { 05:34:48 DEBUG "pod": "ds-cts-0" 05:34:48 DEBUG }, 05:34:48 DEBUG "value": [ 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG [...] 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG 05:34:48 DEBUG "pod": "ds-cts-2" 05:34:48 DEBUG }, 05:34:48 DEBUG "value": [ 05:34:48 DEBUG 1713760488, 05:34:48 DEBUG "32.733333333333334" 05:34:48 DEBUG ] 05:34:48 DEBUG } 05:34:48 DEBUG ] 05:34:48 DEBUG } 05:34:48 DEBUG } 05:34:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760489" 05:34:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760489" 05:34:49 INFO [http_cmd]: http status code OK 05:34:49 DEBUG --- status code --- 05:34:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:49 DEBUG --- http response --- 05:34:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:49 DEBUG ----- output ----- 05:34:49 DEBUG { 05:34:49 DEBUG "status": "success", 05:34:49 DEBUG "data": { 05:34:49 DEBUG "resultType": "vector", 05:34:49 DEBUG "result": [ 05:34:49 DEBUG { 05:34:49 DEBUG "metric": { 05:34:49 DEBUG "pod": "ds-idrepo-0" 05:34:49 DEBUG }, 05:34:49 DEBUG "value": [ 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG [...] 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 INFO [http_cmd]: http status code OK 05:34:49 DEBUG 05:34:49 DEBUG --- status code --- 05:34:49 DEBUG "pod": "ds-idrepo-2" 05:34:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:49 DEBUG }, 05:34:49 DEBUG --- http response --- 05:34:49 DEBUG "value": [ 05:34:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:49 DEBUG 1713760489, 05:34:49 DEBUG ----- output ----- 05:34:49 DEBUG "0" 05:34:49 DEBUG { 05:34:49 DEBUG ] 05:34:49 DEBUG "status": "success", 05:34:49 DEBUG } 05:34:49 DEBUG "data": { 05:34:49 DEBUG ] 05:34:49 DEBUG "resultType": "vector", 05:34:49 DEBUG } 05:34:49 DEBUG "result": [ 05:34:49 DEBUG } 05:34:49 DEBUG { 05:34:49 DEBUG "metric": { 05:34:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:49 DEBUG }, 05:34:49 DEBUG "value": [ 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG [...] 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:49 DEBUG }, 05:34:49 DEBUG "value": [ 05:34:49 DEBUG 1713760489, 05:34:49 DEBUG "972.29" 05:34:49 DEBUG ] 05:34:49 DEBUG } 05:34:49 DEBUG ] 05:34:49 DEBUG } 05:34:49 DEBUG } 05:34:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760489" 05:34:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760489" 05:34:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760489" 05:34:49 INFO [http_cmd]: http status code OK 05:34:49 INFO [http_cmd]: http status code OK 05:34:49 INFO [http_cmd]: http status code OK 05:34:49 DEBUG --- status code --- 05:34:49 DEBUG --- status code --- 05:34:49 DEBUG --- status code --- 05:34:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:49 DEBUG --- http response --- 05:34:49 DEBUG --- http response --- 05:34:49 DEBUG --- http response --- 05:34:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:49 DEBUG ----- output ----- 05:34:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:34:49 DEBUG ----- output ----- 05:34:49 DEBUG { 05:34:49 DEBUG ----- output ----- 05:34:49 DEBUG { 05:34:49 DEBUG "status": "success", 05:34:49 DEBUG { 05:34:49 DEBUG "status": "success", 05:34:49 DEBUG "data": { 05:34:49 DEBUG "status": "success", 05:34:49 DEBUG "data": { 05:34:49 DEBUG "resultType": "vector", 05:34:49 DEBUG "data": { 05:34:49 DEBUG "resultType": "vector", 05:34:49 DEBUG "result": [ 05:34:49 DEBUG "resultType": "vector", 05:34:49 DEBUG "result": [ 05:34:49 DEBUG { 05:34:49 DEBUG "result": [ 05:34:49 DEBUG { 05:34:49 DEBUG "metric": { 05:34:49 DEBUG { 05:34:49 DEBUG "metric": { 05:34:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:49 DEBUG "metric": { 05:34:49 DEBUG "instance": "10.142.0.107:9100" 05:34:49 DEBUG }, 05:34:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:34:49 DEBUG }, 05:34:49 DEBUG "value": [ 05:34:49 DEBUG }, 05:34:49 DEBUG "value": [ 05:34:49 DEBUG 05:34:49 DEBUG "value": [ 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG [...] 05:34:49 DEBUG 05:34:49 DEBUG [...] 05:34:49 DEBUG 05:34:49 DEBUG [...] 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG 05:34:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:49 DEBUG 05:34:49 DEBUG "instance": "10.142.0.43:9100" 05:34:49 DEBUG }, 05:34:49 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:34:49 DEBUG }, 05:34:49 DEBUG "value": [ 05:34:49 DEBUG }, 05:34:49 DEBUG "value": [ 05:34:49 DEBUG 1713760489, 05:34:49 DEBUG "value": [ 05:34:49 DEBUG 1713760489, 05:34:49 DEBUG "0" 05:34:49 DEBUG 1713760489, 05:34:49 DEBUG "0.524800000000107" 05:34:49 DEBUG ] 05:34:49 DEBUG "0" 05:34:49 DEBUG ] 05:34:49 DEBUG } 05:34:49 DEBUG ] 05:34:49 DEBUG } 05:34:49 DEBUG ] 05:34:49 DEBUG } 05:34:49 DEBUG ] 05:34:49 DEBUG } 05:34:49 DEBUG ] 05:34:49 DEBUG } 05:34:49 DEBUG } 05:34:49 DEBUG } 05:34:49 DEBUG } 05:34:49 DEBUG } 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760490" 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG { 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG "data": { 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG "result": [ 05:34:50 DEBUG { 05:34:50 DEBUG "metric": { 05:34:50 DEBUG "instance": "10.142.0.107:9100" 05:34:50 DEBUG }, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG [...] 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG "instance": "10.142.0.43:9100" 05:34:50 DEBUG }, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG "4841800.633333334" 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG { 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG "data": { 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG "result": [ 05:34:50 DEBUG { 05:34:50 DEBUG "metric": { 05:34:50 DEBUG "instance": "10.142.0.107:9100" 05:34:50 DEBUG }, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG [...] 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG "instance": "10.142.0.43:9100" 05:34:50 DEBUG }, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG "0.0769999999999963" 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760490" 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG { 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG "data": { 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG "result": [ 05:34:50 DEBUG { 05:34:50 DEBUG "metric": { 05:34:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:34:50 DEBUG }, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG [...] 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:34:50 DEBUG }, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG "4553195520" 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760490" 05:34:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760490" 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG --- status code --- 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 DEBUG --- status code --- 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG --- http response --- 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:50 DEBUG ----- output ----- 05:34:50 INFO [http_cmd]: http status code OK 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:50 DEBUG { 05:34:50 DEBUG --- status code --- 05:34:50 DEBUG { 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG { 05:34:50 DEBUG { 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG { 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG { 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG { 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG { 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG { 05:34:50 DEBUG "data": { 05:34:50 DEBUG --- http response --- 05:34:50 DEBUG "data": { 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG "data": { 05:34:50 DEBUG "data": { 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG "data": { 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG "data": { 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG { 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG "data": { 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG "result": [ 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG "data": { 05:34:50 DEBUG "result": [ 05:34:50 DEBUG ----- output ----- 05:34:50 DEBUG "result": [ 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG "result": [ 05:34:50 DEBUG "result": [ 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG { 05:34:50 DEBUG "result": [ 05:34:50 DEBUG "data": { 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG { 05:34:50 DEBUG { 05:34:50 DEBUG { 05:34:50 DEBUG "metric": { 05:34:50 DEBUG { 05:34:50 DEBUG { 05:34:50 DEBUG "result": [ 05:34:50 DEBUG "metric": { 05:34:50 DEBUG { 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG "result": [ 05:34:50 DEBUG "metric": { 05:34:50 DEBUG "status": "success", 05:34:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:50 DEBUG "data": { 05:34:50 DEBUG "metric": { 05:34:50 DEBUG "metric": { 05:34:50 DEBUG { 05:34:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:50 DEBUG "metric": { 05:34:50 DEBUG "result": [ 05:34:50 DEBUG { 05:34:50 DEBUG "instance": "10.142.0.107:9100" 05:34:50 DEBUG "data": { 05:34:50 DEBUG }, 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG "result": [ 05:34:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:50 DEBUG "metric": { 05:34:50 DEBUG }, 05:34:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:50 DEBUG { 05:34:50 DEBUG "metric": { 05:34:50 DEBUG }, 05:34:50 DEBUG "resultType": "vector", 05:34:50 DEBUG "value": [ 05:34:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:34:50 DEBUG { 05:34:50 DEBUG }, 05:34:50 DEBUG "pod": "ds-idrepo-0" 05:34:50 DEBUG "value": [ 05:34:50 DEBUG }, 05:34:50 DEBUG "metric": { 05:34:50 DEBUG "pod": "ds-cts-0" 05:34:50 DEBUG "value": [ 05:34:50 DEBUG "result": [ 05:34:50 DEBUG 05:34:50 DEBUG }, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG "value": [ 05:34:50 DEBUG }, 05:34:50 DEBUG 05:34:50 DEBUG "value": [ 05:34:50 DEBUG "pod": "ds-idrepo-0" 05:34:50 DEBUG }, 05:34:50 DEBUG 05:34:50 DEBUG { 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG "value": [ 05:34:50 DEBUG "metric": { 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG }, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 05:34:50 DEBUG "metric": { 05:34:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG }, 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG }, 05:34:50 DEBUG [...] 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG [...] 05:34:50 DEBUG [...] 05:34:50 DEBUG "value": [ 05:34:50 DEBUG [...] 05:34:50 DEBUG [...] 05:34:50 DEBUG 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 05:34:50 DEBUG [...] 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG [...] 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG [...] 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG [...] 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG [...] 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG "instance": "10.142.0.43:9100" 05:34:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:50 DEBUG }, 05:34:50 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:34:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:50 DEBUG 05:34:50 DEBUG 05:34:50 DEBUG }, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG [...] 05:34:50 DEBUG }, 05:34:50 DEBUG }, 05:34:50 DEBUG "pod": "ds-cts-2" 05:34:50 DEBUG 05:34:50 DEBUG }, 05:34:50 DEBUG }, 05:34:50 DEBUG 05:34:50 DEBUG "pod": "ds-cts-2" 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG 05:34:50 DEBUG "value": [ 05:34:50 DEBUG "value": [ 05:34:50 DEBUG }, 05:34:50 DEBUG 05:34:50 DEBUG "value": [ 05:34:50 DEBUG "value": [ 05:34:50 DEBUG "pod": "ds-idrepo-2" 05:34:50 DEBUG }, 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG "4837533.633333333" 05:34:50 DEBUG 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG }, 05:34:50 DEBUG "value": [ 05:34:50 DEBUG "0" 05:34:50 DEBUG ] 05:34:50 DEBUG 05:34:50 DEBUG "0" 05:34:50 DEBUG "0" 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG }, 05:34:50 DEBUG "0" 05:34:50 DEBUG "0" 05:34:50 DEBUG "value": [ 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:34:50 DEBUG ] 05:34:50 DEBUG ] 05:34:50 DEBUG "0" 05:34:50 DEBUG "value": [ 05:34:50 DEBUG ] 05:34:50 DEBUG ] 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG "0" 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG }, 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG "0" 05:34:50 DEBUG ] 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG "value": [ 05:34:50 DEBUG ] 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG "1.003110246654197" 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG 1713760490, 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG ] 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG "1.0311918590623683" 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG } 05:34:50 DEBUG ] 05:34:50 DEBUG } 05:34:50 DEBUG } 05:35:07 INFO 05:35:07 INFO [loop_until]: kubectl --namespace=xlou top pods 05:35:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:35:07 INFO [loop_until]: OK (rc = 0) 05:35:07 DEBUG --- stdout --- 05:35:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 994m 8791Mi am-767f6b67b5-qcqz2 988m 8840Mi am-767f6b67b5-vbltz 965m 8779Mi ds-cts-0 18275m 12506Mi ds-cts-1 18530m 12543Mi ds-cts-2 18450m 12553Mi ds-idrepo-0 14m 3998Mi ds-idrepo-1 14m 4002Mi ds-idrepo-2 13m 3985Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1183Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 8m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 986m 4358Mi 05:35:07 DEBUG --- stderr --- 05:35:07 DEBUG 05:35:15 INFO 05:35:15 INFO [loop_until]: kubectl --namespace=xlou top node 05:35:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:35:15 INFO [loop_until]: OK (rc = 0) 05:35:15 DEBUG --- stdout --- 05:35:15 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6236Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 8081Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1069m 3% 11506Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3345Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 17989m 60% 15762Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5211Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1038m 3% 11725Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 3141Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1137m 3% 11029Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 7484Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 16919m 56% 17713Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 96m 0% 8051Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18363m 61% 15281Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1072m 6% 14336Mi 24% 05:35:15 DEBUG --- stderr --- 05:35:15 DEBUG 05:35:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760547" 05:35:47 ERROR [http_cmd]: ERROR 05:35:47 ERROR --- status code --- 05:35:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:35:47 ERROR --- http response --- 05:35:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:35:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:35:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760547" 05:35:47 INFO [http_cmd]: http status code OK 05:35:47 DEBUG --- status code --- 05:35:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:47 DEBUG --- http response --- 05:35:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760547" 05:35:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760547" 05:35:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760547" 05:35:47 INFO [http_cmd]: http status code OK 05:35:47 INFO [http_cmd]: http status code OK 05:35:47 INFO [http_cmd]: http status code OK 05:35:47 DEBUG --- status code --- 05:35:47 DEBUG --- status code --- 05:35:47 DEBUG --- status code --- 05:35:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:47 DEBUG --- http response --- 05:35:47 DEBUG --- http response --- 05:35:47 DEBUG --- http response --- 05:35:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:35:47 DEBUG ----- output ----- 05:35:47 DEBUG ----- output ----- 05:35:47 DEBUG ----- output ----- 05:35:47 DEBUG { 05:35:47 DEBUG { 05:35:47 DEBUG { 05:35:47 DEBUG "status": "success", 05:35:47 DEBUG "status": "success", 05:35:47 DEBUG "status": "success", 05:35:47 DEBUG "data": { 05:35:47 DEBUG "data": { 05:35:47 DEBUG "data": { 05:35:47 DEBUG "resultType": "vector", 05:35:47 DEBUG "resultType": "vector", 05:35:47 DEBUG "resultType": "vector", 05:35:47 DEBUG "result": [ 05:35:47 DEBUG "result": [ 05:35:47 DEBUG "result": [ 05:35:47 DEBUG { 05:35:47 DEBUG { 05:35:47 DEBUG { 05:35:47 DEBUG "metric": { 05:35:47 DEBUG "metric": { 05:35:47 DEBUG "metric": { 05:35:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:35:47 DEBUG "pod": "ds-idrepo-0" 05:35:47 DEBUG }, 05:35:47 DEBUG }, 05:35:47 DEBUG "pod": "ds-cts-0" 05:35:47 DEBUG "value": [ 05:35:47 DEBUG "value": [ 05:35:47 DEBUG }, 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG "value": [ 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG [...] 05:35:47 DEBUG [...] 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG [...] 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG 05:35:47 DEBUG "pod": "ds-idrepo-2" 05:35:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:47 DEBUG 05:35:47 DEBUG }, 05:35:47 DEBUG }, 05:35:47 DEBUG "pod": "ds-cts-2" 05:35:47 DEBUG "value": [ 05:35:47 DEBUG "value": [ 05:35:47 DEBUG }, 05:35:47 DEBUG 1713760547, 05:35:47 DEBUG 1713760547, 05:35:47 DEBUG "value": [ 05:35:47 DEBUG "0" 05:35:47 DEBUG "0" 05:35:47 DEBUG 1713760547, 05:35:47 DEBUG ] 05:35:47 DEBUG ] 05:35:47 DEBUG "93.2812" 05:35:47 DEBUG } 05:35:47 DEBUG } 05:35:47 DEBUG ] 05:35:47 DEBUG ] 05:35:47 DEBUG ] 05:35:47 DEBUG } 05:35:47 DEBUG } 05:35:47 DEBUG } 05:35:47 DEBUG ] 05:35:47 DEBUG } 05:35:47 DEBUG } 05:35:47 DEBUG } 05:35:47 DEBUG } 05:35:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760548" 05:35:48 INFO [http_cmd]: http status code OK 05:35:48 DEBUG --- status code --- 05:35:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:48 DEBUG --- http response --- 05:35:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:48 DEBUG ----- output ----- 05:35:48 DEBUG { 05:35:48 DEBUG "status": "success", 05:35:48 DEBUG "data": { 05:35:48 DEBUG "resultType": "vector", 05:35:48 DEBUG "result": [ 05:35:48 DEBUG { 05:35:48 DEBUG "metric": { 05:35:48 DEBUG "instance": "10.142.0.107:9100" 05:35:48 DEBUG }, 05:35:48 DEBUG "value": [ 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG [...] 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG "instance": "10.142.0.43:9100" 05:35:48 DEBUG }, 05:35:48 DEBUG "value": [ 05:35:48 DEBUG 1713760548, 05:35:48 DEBUG "5815329068032" 05:35:48 DEBUG ] 05:35:48 DEBUG } 05:35:48 DEBUG ] 05:35:48 DEBUG } 05:35:48 DEBUG } 05:35:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760548" 05:35:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760548" 05:35:48 INFO [http_cmd]: http status code OK 05:35:48 DEBUG --- status code --- 05:35:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:48 DEBUG --- http response --- 05:35:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:48 DEBUG ----- output ----- 05:35:48 DEBUG { 05:35:48 DEBUG "status": "success", 05:35:48 DEBUG "data": { 05:35:48 DEBUG "resultType": "vector", 05:35:48 DEBUG "result": [ 05:35:48 DEBUG { 05:35:48 DEBUG "metric": { 05:35:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:48 DEBUG }, 05:35:48 DEBUG "value": [ 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG [...] 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:48 DEBUG }, 05:35:48 DEBUG "value": [ 05:35:48 DEBUG 1713760548, 05:35:48 DEBUG "1000" 05:35:48 DEBUG ] 05:35:48 DEBUG } 05:35:48 DEBUG ] 05:35:48 DEBUG } 05:35:48 DEBUG } 05:35:48 INFO [http_cmd]: http status code OK 05:35:48 DEBUG --- status code --- 05:35:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:48 DEBUG --- http response --- 05:35:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:48 DEBUG ----- output ----- 05:35:48 DEBUG { 05:35:48 DEBUG "status": "success", 05:35:48 DEBUG "data": { 05:35:48 DEBUG "resultType": "vector", 05:35:48 DEBUG "result": [ 05:35:48 DEBUG { 05:35:48 DEBUG "metric": { 05:35:48 DEBUG "pod": "ds-cts-0" 05:35:48 DEBUG }, 05:35:48 DEBUG "value": [ 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG [...] 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG 05:35:48 DEBUG "pod": "ds-cts-2" 05:35:48 DEBUG }, 05:35:48 DEBUG "value": [ 05:35:48 DEBUG 1713760548, 05:35:48 DEBUG "35.9" 05:35:48 DEBUG ] 05:35:48 DEBUG } 05:35:48 DEBUG ] 05:35:48 DEBUG } 05:35:48 DEBUG } 05:35:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760549" 05:35:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760549" 05:35:49 INFO [http_cmd]: http status code OK 05:35:49 INFO [http_cmd]: http status code OK 05:35:49 DEBUG --- status code --- 05:35:49 DEBUG --- status code --- 05:35:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:49 DEBUG --- http response --- 05:35:49 DEBUG --- http response --- 05:35:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:49 DEBUG ----- output ----- 05:35:49 DEBUG ----- output ----- 05:35:49 DEBUG { 05:35:49 DEBUG { 05:35:49 DEBUG "status": "success", 05:35:49 DEBUG "status": "success", 05:35:49 DEBUG "data": { 05:35:49 DEBUG "data": { 05:35:49 DEBUG "resultType": "vector", 05:35:49 DEBUG "resultType": "vector", 05:35:49 DEBUG "result": [ 05:35:49 DEBUG "result": [ 05:35:49 DEBUG { 05:35:49 DEBUG { 05:35:49 DEBUG "metric": { 05:35:49 DEBUG "metric": { 05:35:49 DEBUG "pod": "ds-idrepo-0" 05:35:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:49 DEBUG }, 05:35:49 DEBUG }, 05:35:49 DEBUG "value": [ 05:35:49 DEBUG "value": [ 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG [...] 05:35:49 DEBUG [...] 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG "pod": "ds-idrepo-2" 05:35:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:49 DEBUG }, 05:35:49 DEBUG }, 05:35:49 DEBUG "value": [ 05:35:49 DEBUG "value": [ 05:35:49 DEBUG 1713760549, 05:35:49 DEBUG 1713760549, 05:35:49 DEBUG "0" 05:35:49 DEBUG "974.6890000000001" 05:35:49 DEBUG ] 05:35:49 DEBUG ] 05:35:49 DEBUG } 05:35:49 DEBUG ] 05:35:49 DEBUG } 05:35:49 DEBUG } 05:35:49 DEBUG } 05:35:49 DEBUG ] 05:35:49 DEBUG } 05:35:49 DEBUG } 05:35:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760549" 05:35:49 INFO [http_cmd]: http status code OK 05:35:49 DEBUG --- status code --- 05:35:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:49 DEBUG --- http response --- 05:35:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:49 DEBUG ----- output ----- 05:35:49 DEBUG { 05:35:49 DEBUG "status": "success", 05:35:49 DEBUG "data": { 05:35:49 DEBUG "resultType": "vector", 05:35:49 DEBUG "result": [ 05:35:49 DEBUG { 05:35:49 DEBUG "metric": { 05:35:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:35:49 DEBUG }, 05:35:49 DEBUG "value": [ 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG [...] 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:35:49 DEBUG }, 05:35:49 DEBUG "value": [ 05:35:49 DEBUG 1713760549, 05:35:49 DEBUG "133.47671652491283" 05:35:49 DEBUG ] 05:35:49 DEBUG } 05:35:49 DEBUG ] 05:35:49 DEBUG } 05:35:49 DEBUG } 05:35:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760549" 05:35:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760549" 05:35:49 INFO [http_cmd]: http status code OK 05:35:49 DEBUG --- status code --- 05:35:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:49 DEBUG --- http response --- 05:35:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:49 DEBUG ----- output ----- 05:35:49 DEBUG { 05:35:49 DEBUG "status": "success", 05:35:49 DEBUG "data": { 05:35:49 DEBUG "resultType": "vector", 05:35:49 DEBUG "result": [ 05:35:49 DEBUG { 05:35:49 DEBUG "metric": { 05:35:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:49 DEBUG }, 05:35:49 DEBUG "value": [ 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG [...] 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:49 DEBUG }, 05:35:49 DEBUG "value": [ 05:35:49 DEBUG 1713760549, 05:35:49 DEBUG "0" 05:35:49 DEBUG ] 05:35:49 DEBUG } 05:35:49 DEBUG ] 05:35:49 DEBUG } 05:35:49 DEBUG } 05:35:49 INFO [http_cmd]: http status code OK 05:35:49 DEBUG --- status code --- 05:35:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:49 DEBUG --- http response --- 05:35:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:49 DEBUG ----- output ----- 05:35:49 DEBUG { 05:35:49 DEBUG "status": "success", 05:35:49 DEBUG "data": { 05:35:49 DEBUG "resultType": "vector", 05:35:49 DEBUG "result": [ 05:35:49 DEBUG { 05:35:49 DEBUG "metric": { 05:35:49 DEBUG "instance": "10.142.0.107:9100" 05:35:49 DEBUG }, 05:35:49 DEBUG "value": [ 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG [...] 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG 05:35:49 DEBUG "instance": "10.142.0.43:9100" 05:35:49 DEBUG }, 05:35:49 DEBUG "value": [ 05:35:49 DEBUG 1713760549, 05:35:49 DEBUG "0.5064666666665345" 05:35:49 DEBUG ] 05:35:49 DEBUG } 05:35:49 DEBUG ] 05:35:49 DEBUG } 05:35:49 DEBUG } 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760550" 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 DEBUG --- status code --- 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG { 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG "data": { 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "result": [ 05:35:50 DEBUG { 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "instance": "10.142.0.107:9100" 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG "instance": "10.142.0.43:9100" 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG "4851275.233333333" 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 DEBUG --- status code --- 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG { 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG "data": { 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "result": [ 05:35:50 DEBUG { 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "instance": "10.142.0.107:9100" 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG "instance": "10.142.0.43:9100" 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG "0.06766666666665817" 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760550" 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 DEBUG --- status code --- 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG { 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG "data": { 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "result": [ 05:35:50 DEBUG { 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG "4507394048" 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760550" 05:35:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760550" 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 DEBUG --- status code --- 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 DEBUG --- status code --- 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 DEBUG --- status code --- 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 DEBUG --- status code --- 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 DEBUG --- status code --- 05:35:50 DEBUG --- status code --- 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG --- status code --- 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG --- status code --- 05:35:50 INFO [http_cmd]: http status code OK 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG --- status code --- 05:35:50 DEBUG --- status code --- 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG --- status code --- 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG { 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG { 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG { 05:35:50 DEBUG { 05:35:50 DEBUG --- http response --- 05:35:50 DEBUG { 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG { 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG { 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG { 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG { 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG "data": { 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG "data": { 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG "data": { 05:35:50 DEBUG "data": { 05:35:50 DEBUG ----- output ----- 05:35:50 DEBUG "data": { 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG "data": { 05:35:50 DEBUG { 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "data": { 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "data": { 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG { 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "data": { 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG "result": [ 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "result": [ 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "result": [ 05:35:50 DEBUG "result": [ 05:35:50 DEBUG "status": "success", 05:35:50 DEBUG "result": [ 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "result": [ 05:35:50 DEBUG "data": { 05:35:50 DEBUG { 05:35:50 DEBUG "result": [ 05:35:50 DEBUG { 05:35:50 DEBUG "result": [ 05:35:50 DEBUG { 05:35:50 DEBUG { 05:35:50 DEBUG "data": { 05:35:50 DEBUG { 05:35:50 DEBUG "result": [ 05:35:50 DEBUG { 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "metric": { 05:35:50 DEBUG { 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "pod": "ds-idrepo-0" 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "resultType": "vector", 05:35:50 DEBUG "result": [ 05:35:50 DEBUG { 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "result": [ 05:35:50 DEBUG "pod": "ds-idrepo-1" 05:35:50 DEBUG { 05:35:50 DEBUG }, 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:50 DEBUG { 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "pod": "ds-idrepo-0" 05:35:50 DEBUG { 05:35:50 DEBUG }, 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:50 DEBUG }, 05:35:50 DEBUG }, 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "instance": "10.142.0.107:9100" 05:35:50 DEBUG "pod": "ds-idrepo-1" 05:35:50 DEBUG }, 05:35:50 DEBUG "metric": { 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:50 DEBUG 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "pod": "ds-cts-0" 05:35:50 DEBUG }, 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:35:50 DEBUG 05:35:50 DEBUG }, 05:35:50 DEBUG 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG }, 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG "pod": "ds-cts-2" 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG "pod": "ds-idrepo-2" 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG 05:35:50 DEBUG [...] 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:35:50 DEBUG 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG 05:35:50 DEBUG }, 05:35:50 DEBUG }, 05:35:50 DEBUG 05:35:50 DEBUG "instance": "10.142.0.43:9100" 05:35:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:35:50 DEBUG 05:35:50 DEBUG "0" 05:35:50 DEBUG }, 05:35:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:50 DEBUG "0" 05:35:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "pod": "ds-cts-2" 05:35:50 DEBUG }, 05:35:50 DEBUG }, 05:35:50 DEBUG 05:35:50 DEBUG ] 05:35:50 DEBUG "value": [ 05:35:50 DEBUG }, 05:35:50 DEBUG ] 05:35:50 DEBUG }, 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG }, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 05:35:50 DEBUG } 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG "value": [ 05:35:50 DEBUG } 05:35:50 DEBUG "value": [ 05:35:50 DEBUG "0" 05:35:50 DEBUG "0" 05:35:50 DEBUG "value": [ 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:35:50 DEBUG }, 05:35:50 DEBUG "0" 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG ] 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG ] 05:35:50 DEBUG ] 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG "4846083.266666667" 05:35:50 DEBUG "0.00001896913628238786" 05:35:50 DEBUG "value": [ 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG "0" 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG "0" 05:35:50 DEBUG ] 05:35:50 DEBUG ] 05:35:50 DEBUG 1713760550, 05:35:50 DEBUG ] 05:35:50 DEBUG "0" 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG ] 05:35:50 DEBUG ] 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG "1.0650593468369298" 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG ] 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG ] 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG } 05:35:50 DEBUG } 05:36:07 INFO 05:36:07 INFO [loop_until]: kubectl --namespace=xlou top pods 05:36:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:36:07 INFO [loop_until]: OK (rc = 0) 05:36:07 DEBUG --- stdout --- 05:36:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 968m 8792Mi am-767f6b67b5-qcqz2 955m 8839Mi am-767f6b67b5-vbltz 1009m 8166Mi ds-cts-0 17424m 12511Mi ds-cts-1 17632m 12546Mi ds-cts-2 18431m 12544Mi ds-idrepo-0 15m 3998Mi ds-idrepo-1 12m 4002Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 427m 4205Mi 05:36:07 DEBUG --- stderr --- 05:36:07 DEBUG 05:36:15 INFO 05:36:15 INFO [loop_until]: kubectl --namespace=xlou top node 05:36:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:36:15 INFO [loop_until]: OK (rc = 0) 05:36:15 DEBUG --- stdout --- 05:36:15 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6236Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 8089Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1006m 3% 11512Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17899m 59% 15765Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5188Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1027m 3% 11737Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 76m 0% 3123Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1121m 3% 10414Mi 9% gke-xlou-cdm-ds-0f601751-j488 67m 0% 7478Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17525m 58% 17718Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 68m 0% 8044Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18364m 61% 15281Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 454m 2% 14164Mi 24% 05:36:15 DEBUG --- stderr --- 05:36:15 DEBUG 05:36:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760607" 05:36:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760607" 05:36:47 ERROR [http_cmd]: ERROR 05:36:47 ERROR --- status code --- 05:36:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:36:47 ERROR --- http response --- 05:36:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:36:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:36:47 INFO [http_cmd]: http status code OK 05:36:47 DEBUG --- status code --- 05:36:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:47 DEBUG --- http response --- 05:36:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760607" 05:36:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760607" 05:36:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760607" 05:36:47 INFO [http_cmd]: http status code OK 05:36:47 DEBUG --- status code --- 05:36:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:47 DEBUG --- http response --- 05:36:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:47 DEBUG ----- output ----- 05:36:47 DEBUG { 05:36:47 DEBUG "status": "success", 05:36:47 DEBUG "data": { 05:36:47 DEBUG "resultType": "vector", 05:36:47 DEBUG "result": [ 05:36:47 DEBUG { 05:36:47 DEBUG "metric": { 05:36:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:36:47 DEBUG }, 05:36:47 DEBUG "value": [ 05:36:47 DEBUG 05:36:47 DEBUG 05:36:47 DEBUG 05:36:47 DEBUG [...] 05:36:47 DEBUG 05:36:47 DEBUG 05:36:47 DEBUG 05:36:47 DEBUG "pod": "ds-cts-2" 05:36:47 DEBUG }, 05:36:47 DEBUG "value": [ 05:36:47 DEBUG 1713760607, 05:36:47 DEBUG "70657.3097797905" 05:36:47 DEBUG ] 05:36:47 DEBUG } 05:36:47 DEBUG ] 05:36:47 DEBUG } 05:36:47 DEBUG } 05:36:47 INFO [http_cmd]: http status code OK 05:36:47 DEBUG --- status code --- 05:36:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:47 DEBUG --- http response --- 05:36:47 INFO [http_cmd]: http status code OK 05:36:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:47 DEBUG --- status code --- 05:36:47 DEBUG ----- output ----- 05:36:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:47 DEBUG { 05:36:47 DEBUG --- http response --- 05:36:47 DEBUG "status": "success", 05:36:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:47 DEBUG "data": { 05:36:47 DEBUG ----- output ----- 05:36:47 DEBUG "resultType": "vector", 05:36:47 DEBUG { 05:36:47 DEBUG "result": [ 05:36:47 DEBUG "status": "success", 05:36:47 DEBUG { 05:36:47 DEBUG "data": { 05:36:47 DEBUG "metric": { 05:36:47 DEBUG "resultType": "vector", 05:36:47 DEBUG "pod": "ds-idrepo-0" 05:36:47 DEBUG "result": [ 05:36:47 DEBUG }, 05:36:47 DEBUG { 05:36:47 DEBUG "value": [ 05:36:47 DEBUG "metric": { 05:36:47 DEBUG 05:36:47 DEBUG "pod": "ds-cts-0" 05:36:47 DEBUG 05:36:47 DEBUG }, 05:36:47 DEBUG 05:36:47 DEBUG "value": [ 05:36:47 DEBUG [...] 05:36:47 DEBUG 05:36:47 DEBUG 05:36:47 DEBUG 05:36:47 DEBUG 05:36:47 DEBUG 05:36:47 DEBUG 05:36:47 DEBUG [...] 05:36:47 DEBUG "pod": "ds-idrepo-2" 05:36:47 DEBUG 05:36:47 DEBUG }, 05:36:47 DEBUG 05:36:47 DEBUG "value": [ 05:36:47 DEBUG 05:36:47 DEBUG 1713760607, 05:36:47 DEBUG "pod": "ds-cts-2" 05:36:47 DEBUG "0" 05:36:47 DEBUG ] 05:36:47 DEBUG } 05:36:47 DEBUG ] 05:36:47 DEBUG } 05:36:47 DEBUG } 05:36:47 DEBUG }, 05:36:47 DEBUG "value": [ 05:36:47 DEBUG 1713760607, 05:36:47 DEBUG "91.0909" 05:36:47 DEBUG ] 05:36:47 DEBUG } 05:36:47 DEBUG ] 05:36:47 DEBUG } 05:36:47 DEBUG } 05:36:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760608" 05:36:48 INFO [http_cmd]: http status code OK 05:36:48 DEBUG --- status code --- 05:36:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:48 DEBUG --- http response --- 05:36:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:48 DEBUG ----- output ----- 05:36:48 DEBUG { 05:36:48 DEBUG "status": "success", 05:36:48 DEBUG "data": { 05:36:48 DEBUG "resultType": "vector", 05:36:48 DEBUG "result": [ 05:36:48 DEBUG { 05:36:48 DEBUG "metric": { 05:36:48 DEBUG "instance": "10.142.0.107:9100" 05:36:48 DEBUG }, 05:36:48 DEBUG "value": [ 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG [...] 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG "instance": "10.142.0.43:9100" 05:36:48 DEBUG }, 05:36:48 DEBUG "value": [ 05:36:48 DEBUG 1713760608, 05:36:48 DEBUG "5819347526656" 05:36:48 DEBUG ] 05:36:48 DEBUG } 05:36:48 DEBUG ] 05:36:48 DEBUG } 05:36:48 DEBUG } 05:36:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760608" 05:36:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760608" 05:36:48 INFO [http_cmd]: http status code OK 05:36:48 INFO [http_cmd]: http status code OK 05:36:48 DEBUG --- status code --- 05:36:48 DEBUG --- status code --- 05:36:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:48 DEBUG --- http response --- 05:36:48 DEBUG --- http response --- 05:36:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:48 DEBUG ----- output ----- 05:36:48 DEBUG ----- output ----- 05:36:48 DEBUG { 05:36:48 DEBUG { 05:36:48 DEBUG "status": "success", 05:36:48 DEBUG "status": "success", 05:36:48 DEBUG "data": { 05:36:48 DEBUG "data": { 05:36:48 DEBUG "resultType": "vector", 05:36:48 DEBUG "resultType": "vector", 05:36:48 DEBUG "result": [ 05:36:48 DEBUG "result": [ 05:36:48 DEBUG { 05:36:48 DEBUG { 05:36:48 DEBUG "metric": { 05:36:48 DEBUG "metric": { 05:36:48 DEBUG "pod": "ds-cts-0" 05:36:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:48 DEBUG }, 05:36:48 DEBUG }, 05:36:48 DEBUG "value": [ 05:36:48 DEBUG "value": [ 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG [...] 05:36:48 DEBUG [...] 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG 05:36:48 DEBUG "pod": "ds-cts-2" 05:36:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:48 DEBUG }, 05:36:48 DEBUG }, 05:36:48 DEBUG "value": [ 05:36:48 DEBUG "value": [ 05:36:48 DEBUG 1713760608, 05:36:48 DEBUG 1713760608, 05:36:48 DEBUG "37.4" 05:36:48 DEBUG "1000" 05:36:48 DEBUG ] 05:36:48 DEBUG } 05:36:48 DEBUG ] 05:36:48 DEBUG } 05:36:48 DEBUG } 05:36:48 DEBUG ] 05:36:48 DEBUG } 05:36:48 DEBUG ] 05:36:48 DEBUG } 05:36:48 DEBUG } 05:36:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760609" 05:36:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760609" 05:36:49 INFO [http_cmd]: http status code OK 05:36:49 INFO [http_cmd]: http status code OK 05:36:49 DEBUG --- status code --- 05:36:49 DEBUG --- status code --- 05:36:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:49 DEBUG --- http response --- 05:36:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:49 DEBUG --- http response --- 05:36:49 DEBUG ----- output ----- 05:36:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:49 DEBUG { 05:36:49 DEBUG ----- output ----- 05:36:49 DEBUG "status": "success", 05:36:49 DEBUG { 05:36:49 DEBUG "data": { 05:36:49 DEBUG "status": "success", 05:36:49 DEBUG "resultType": "vector", 05:36:49 DEBUG "data": { 05:36:49 DEBUG "result": [ 05:36:49 DEBUG "resultType": "vector", 05:36:49 DEBUG "result": [ 05:36:49 DEBUG { 05:36:49 DEBUG "metric": { 05:36:49 DEBUG "pod": "ds-idrepo-0" 05:36:49 DEBUG }, 05:36:49 DEBUG "value": [ 05:36:49 DEBUG 05:36:49 DEBUG { 05:36:49 DEBUG 05:36:49 DEBUG "metric": { 05:36:49 DEBUG 05:36:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:49 DEBUG [...] 05:36:49 DEBUG }, 05:36:49 DEBUG 05:36:49 DEBUG "value": [ 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG "pod": "ds-idrepo-2" 05:36:49 DEBUG 05:36:49 DEBUG }, 05:36:49 DEBUG [...] 05:36:49 DEBUG "value": [ 05:36:49 DEBUG 05:36:49 DEBUG 1713760609, 05:36:49 DEBUG 05:36:49 DEBUG "0" 05:36:49 DEBUG 05:36:49 DEBUG ] 05:36:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:49 DEBUG } 05:36:49 DEBUG }, 05:36:49 DEBUG ] 05:36:49 DEBUG "value": [ 05:36:49 DEBUG } 05:36:49 DEBUG 1713760609, 05:36:49 DEBUG } 05:36:49 DEBUG "977.0695000000001" 05:36:49 DEBUG ] 05:36:49 DEBUG } 05:36:49 DEBUG ] 05:36:49 DEBUG } 05:36:49 DEBUG } 05:36:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760609" 05:36:49 INFO [http_cmd]: http status code OK 05:36:49 DEBUG --- status code --- 05:36:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:49 DEBUG --- http response --- 05:36:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:49 DEBUG ----- output ----- 05:36:49 DEBUG { 05:36:49 DEBUG "status": "success", 05:36:49 DEBUG "data": { 05:36:49 DEBUG "resultType": "vector", 05:36:49 DEBUG "result": [ 05:36:49 DEBUG { 05:36:49 DEBUG "metric": { 05:36:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:36:49 DEBUG }, 05:36:49 DEBUG "value": [ 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG [...] 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG "pod": "ds-cts-0" 05:36:49 DEBUG }, 05:36:49 DEBUG "value": [ 05:36:49 DEBUG 1713760609, 05:36:49 DEBUG "68865779.74259332" 05:36:49 DEBUG ] 05:36:49 DEBUG } 05:36:49 DEBUG ] 05:36:49 DEBUG } 05:36:49 DEBUG } 05:36:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760609" 05:36:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760609" 05:36:49 INFO [http_cmd]: http status code OK 05:36:49 DEBUG --- status code --- 05:36:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:49 DEBUG --- http response --- 05:36:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:49 DEBUG ----- output ----- 05:36:49 DEBUG { 05:36:49 DEBUG "status": "success", 05:36:49 DEBUG "data": { 05:36:49 DEBUG "resultType": "vector", 05:36:49 DEBUG "result": [ 05:36:49 DEBUG { 05:36:49 DEBUG "metric": { 05:36:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:49 DEBUG }, 05:36:49 DEBUG "value": [ 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG [...] 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:49 DEBUG }, 05:36:49 DEBUG "value": [ 05:36:49 DEBUG 1713760609, 05:36:49 DEBUG "0" 05:36:49 DEBUG ] 05:36:49 DEBUG } 05:36:49 DEBUG ] 05:36:49 DEBUG } 05:36:49 DEBUG } 05:36:49 INFO [http_cmd]: http status code OK 05:36:49 DEBUG --- status code --- 05:36:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:49 DEBUG --- http response --- 05:36:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:49 DEBUG ----- output ----- 05:36:49 DEBUG { 05:36:49 DEBUG "status": "success", 05:36:49 DEBUG "data": { 05:36:49 DEBUG "resultType": "vector", 05:36:49 DEBUG "result": [ 05:36:49 DEBUG { 05:36:49 DEBUG "metric": { 05:36:49 DEBUG "instance": "10.142.0.107:9100" 05:36:49 DEBUG }, 05:36:49 DEBUG "value": [ 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG [...] 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG 05:36:49 DEBUG "instance": "10.142.0.43:9100" 05:36:49 DEBUG }, 05:36:49 DEBUG "value": [ 05:36:49 DEBUG 1713760609, 05:36:49 DEBUG "0.5176999999999661" 05:36:49 DEBUG ] 05:36:49 DEBUG } 05:36:49 DEBUG ] 05:36:49 DEBUG } 05:36:49 DEBUG } 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760610" 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG { 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG "data": { 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "result": [ 05:36:50 DEBUG { 05:36:50 DEBUG "metric": { 05:36:50 DEBUG "instance": "10.142.0.107:9100" 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG [...] 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG "instance": "10.142.0.43:9100" 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG "4984441.666666667" 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG { 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG "data": { 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "result": [ 05:36:50 DEBUG { 05:36:50 DEBUG "metric": { 05:36:50 DEBUG "instance": "10.142.0.107:9100" 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG [...] 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG "instance": "10.142.0.43:9100" 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG "0.07533333333335197" 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760610" 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG { 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG "data": { 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "result": [ 05:36:50 DEBUG { 05:36:50 DEBUG "metric": { 05:36:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG [...] 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG "4870832128" 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760610" 05:36:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760610" 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 DEBUG --- status code --- 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG --- status code --- 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 INFO [http_cmd]: http status code OK 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG --- status code --- 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:36:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG --- http response --- 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG { 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:50 DEBUG { 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:50 DEBUG { 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG { 05:36:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG { 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG { 05:36:50 DEBUG ----- output ----- 05:36:50 DEBUG { 05:36:50 DEBUG { 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG "data": { 05:36:50 DEBUG { 05:36:50 DEBUG "data": { 05:36:50 DEBUG { 05:36:50 DEBUG "data": { 05:36:50 DEBUG "data": { 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG { 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG "data": { 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "data": { 05:36:50 DEBUG "status": "success", 05:36:50 DEBUG "data": { 05:36:50 DEBUG "data": { 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "result": [ 05:36:50 DEBUG "data": { 05:36:50 DEBUG "result": [ 05:36:50 DEBUG "data": { 05:36:50 DEBUG "result": [ 05:36:50 DEBUG "result": [ 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "data": { 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "result": [ 05:36:50 DEBUG { 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG { 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG { 05:36:50 DEBUG { 05:36:50 DEBUG "result": [ 05:36:50 DEBUG "resultType": "vector", 05:36:50 DEBUG "result": [ 05:36:50 DEBUG "result": [ 05:36:50 DEBUG { 05:36:50 DEBUG "metric": { 05:36:50 DEBUG "result": [ 05:36:50 DEBUG "metric": { 05:36:50 DEBUG "result": [ 05:36:50 DEBUG "metric": { 05:36:50 DEBUG "metric": { 05:36:50 DEBUG { 05:36:50 DEBUG "result": [ 05:36:50 DEBUG { 05:36:50 DEBUG { 05:36:50 DEBUG "metric": { 05:36:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:36:50 DEBUG { 05:36:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:50 DEBUG { 05:36:50 DEBUG "pod": "ds-idrepo-0" 05:36:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:36:50 DEBUG "metric": { 05:36:50 DEBUG { 05:36:50 DEBUG "metric": { 05:36:50 DEBUG "metric": { 05:36:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:50 DEBUG }, 05:36:50 DEBUG "metric": { 05:36:50 DEBUG }, 05:36:50 DEBUG "metric": { 05:36:50 DEBUG }, 05:36:50 DEBUG }, 05:36:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:36:50 DEBUG "metric": { 05:36:50 DEBUG "instance": "10.142.0.107:9100" 05:36:50 DEBUG "pod": "ds-cts-0" 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:50 DEBUG "value": [ 05:36:50 DEBUG "pod": "ds-idrepo-0" 05:36:50 DEBUG "value": [ 05:36:50 DEBUG "value": [ 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG }, 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 05:36:50 DEBUG }, 05:36:50 DEBUG 05:36:50 DEBUG }, 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:50 DEBUG "value": [ 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG }, 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG [...] 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG [...] 05:36:50 DEBUG [...] 05:36:50 DEBUG [...] 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG [...] 05:36:50 DEBUG 05:36:50 DEBUG [...] 05:36:50 DEBUG [...] 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG [...] 05:36:50 DEBUG [...] 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG [...] 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG [...] 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:50 DEBUG "pod": "ds-cts-2" 05:36:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:36:50 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:36:50 DEBUG }, 05:36:50 DEBUG 05:36:50 DEBUG 05:36:50 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:36:50 DEBUG }, 05:36:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:50 DEBUG 05:36:50 DEBUG }, 05:36:50 DEBUG }, 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 05:36:50 DEBUG "instance": "10.142.0.43:9100" 05:36:50 DEBUG "pod": "ds-cts-2" 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG }, 05:36:50 DEBUG 05:36:50 DEBUG "value": [ 05:36:50 DEBUG "value": [ 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG 05:36:50 DEBUG }, 05:36:50 DEBUG }, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG "value": [ 05:36:50 DEBUG "pod": "ds-idrepo-2" 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG "0.7789767194974867" 05:36:50 DEBUG 05:36:50 DEBUG "value": [ 05:36:50 DEBUG "value": [ 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG "0" 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG }, 05:36:50 DEBUG "0" 05:36:50 DEBUG "0" 05:36:50 DEBUG "1.120544760748737" 05:36:50 DEBUG ] 05:36:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG "64208311.78234275" 05:36:50 DEBUG ] 05:36:50 DEBUG "0" 05:36:50 DEBUG "value": [ 05:36:50 DEBUG ] 05:36:50 DEBUG ] 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG }, 05:36:50 DEBUG "4978129.3" 05:36:50 DEBUG "0" 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG "value": [ 05:36:50 DEBUG ] 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG "0" 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG 1713760610, 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG "0" 05:36:50 DEBUG ] 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG ] 05:36:50 DEBUG } 05:36:50 DEBUG } 05:36:50 DEBUG } 05:37:07 INFO 05:37:07 INFO [loop_until]: kubectl --namespace=xlou top pods 05:37:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:37:07 INFO [loop_until]: OK (rc = 0) 05:37:07 DEBUG --- stdout --- 05:37:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 979m 8791Mi am-767f6b67b5-qcqz2 958m 8840Mi am-767f6b67b5-vbltz 996m 8166Mi ds-cts-0 18080m 12511Mi ds-cts-1 17908m 12550Mi ds-cts-2 18582m 12547Mi ds-idrepo-0 14m 4000Mi ds-idrepo-1 33m 4000Mi ds-idrepo-2 12m 3985Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1024m 4812Mi 05:37:07 DEBUG --- stderr --- 05:37:07 DEBUG 05:37:15 INFO 05:37:15 INFO [loop_until]: kubectl --namespace=xlou top node 05:37:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:37:15 INFO [loop_until]: OK (rc = 0) 05:37:15 DEBUG --- stdout --- 05:37:15 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6239Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 8091Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1015m 3% 11531Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3349Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17871m 59% 15749Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5199Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1040m 3% 11737Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 3131Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1099m 3% 10408Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 7474Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17527m 58% 17727Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 110m 0% 8046Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18543m 62% 15267Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1080m 6% 14793Mi 25% 05:37:15 DEBUG --- stderr --- 05:37:15 DEBUG 05:37:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760667" 05:37:47 ERROR [http_cmd]: ERROR 05:37:47 ERROR --- status code --- 05:37:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:37:47 ERROR --- http response --- 05:37:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:37:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:37:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760667" 05:37:47 INFO [http_cmd]: http status code OK 05:37:47 DEBUG --- status code --- 05:37:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:47 DEBUG --- http response --- 05:37:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760667" 05:37:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760667" 05:37:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760667" 05:37:47 INFO [http_cmd]: http status code OK 05:37:47 DEBUG --- status code --- 05:37:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:47 DEBUG --- http response --- 05:37:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:37:47 DEBUG ----- output ----- 05:37:47 DEBUG { 05:37:47 DEBUG "status": "success", 05:37:47 DEBUG "data": { 05:37:47 DEBUG "resultType": "vector", 05:37:47 DEBUG "result": [ 05:37:47 DEBUG { 05:37:47 DEBUG "metric": { 05:37:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:37:47 DEBUG }, 05:37:47 DEBUG "value": [ 05:37:47 DEBUG 05:37:47 DEBUG 05:37:47 DEBUG 05:37:47 DEBUG [...] 05:37:47 DEBUG 05:37:47 DEBUG 05:37:47 DEBUG 05:37:47 DEBUG "pod": "ds-idrepo-0" 05:37:47 DEBUG }, 05:37:47 DEBUG "value": [ 05:37:47 DEBUG 1713760667, 05:37:47 DEBUG "458551.2053718856" 05:37:47 DEBUG ] 05:37:47 DEBUG } 05:37:47 DEBUG ] 05:37:47 DEBUG } 05:37:47 DEBUG } 05:37:47 INFO [http_cmd]: http status code OK 05:37:47 DEBUG --- status code --- 05:37:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:47 INFO [http_cmd]: http status code OK 05:37:47 DEBUG --- http response --- 05:37:47 DEBUG --- status code --- 05:37:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:47 DEBUG ----- output ----- 05:37:47 DEBUG --- http response --- 05:37:47 DEBUG { 05:37:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:47 DEBUG "status": "success", 05:37:47 DEBUG ----- output ----- 05:37:47 DEBUG "data": { 05:37:47 DEBUG { 05:37:47 DEBUG "resultType": "vector", 05:37:47 DEBUG "status": "success", 05:37:47 DEBUG "result": [ 05:37:47 DEBUG "data": { 05:37:47 DEBUG { 05:37:47 DEBUG "resultType": "vector", 05:37:47 DEBUG "metric": { 05:37:47 DEBUG "result": [ 05:37:47 DEBUG "pod": "ds-idrepo-0" 05:37:47 DEBUG { 05:37:47 DEBUG }, 05:37:47 DEBUG "metric": { 05:37:47 DEBUG "value": [ 05:37:47 DEBUG "pod": "ds-cts-0" 05:37:47 DEBUG 05:37:47 DEBUG }, 05:37:47 DEBUG 05:37:47 DEBUG "value": [ 05:37:47 DEBUG 05:37:47 DEBUG 05:37:47 DEBUG [...] 05:37:47 DEBUG 05:37:47 DEBUG 05:37:47 DEBUG 05:37:47 DEBUG 05:37:47 DEBUG [...] 05:37:47 DEBUG 05:37:47 DEBUG 05:37:47 DEBUG "pod": "ds-idrepo-2" 05:37:47 DEBUG 05:37:47 DEBUG }, 05:37:47 DEBUG 05:37:47 DEBUG "value": [ 05:37:47 DEBUG "pod": "ds-cts-2" 05:37:47 DEBUG 1713760667, 05:37:47 DEBUG }, 05:37:47 DEBUG "0" 05:37:47 DEBUG ] 05:37:47 DEBUG "value": [ 05:37:47 DEBUG } 05:37:47 DEBUG 1713760667, 05:37:47 DEBUG ] 05:37:47 DEBUG "89.01946666666666" 05:37:47 DEBUG } 05:37:47 DEBUG ] 05:37:47 DEBUG } 05:37:47 DEBUG } 05:37:47 DEBUG ] 05:37:47 DEBUG } 05:37:47 DEBUG } 05:37:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760668" 05:37:48 INFO [http_cmd]: http status code OK 05:37:48 DEBUG --- status code --- 05:37:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:48 DEBUG --- http response --- 05:37:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:48 DEBUG ----- output ----- 05:37:48 DEBUG { 05:37:48 DEBUG "status": "success", 05:37:48 DEBUG "data": { 05:37:48 DEBUG "resultType": "vector", 05:37:48 DEBUG "result": [ 05:37:48 DEBUG { 05:37:48 DEBUG "metric": { 05:37:48 DEBUG "instance": "10.142.0.107:9100" 05:37:48 DEBUG }, 05:37:48 DEBUG "value": [ 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG [...] 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG "instance": "10.142.0.43:9100" 05:37:48 DEBUG }, 05:37:48 DEBUG "value": [ 05:37:48 DEBUG 1713760668, 05:37:48 DEBUG "5823285242880" 05:37:48 DEBUG ] 05:37:48 DEBUG } 05:37:48 DEBUG ] 05:37:48 DEBUG } 05:37:48 DEBUG } 05:37:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760668" 05:37:48 INFO [http_cmd]: http status code OK 05:37:48 DEBUG --- status code --- 05:37:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:48 DEBUG --- http response --- 05:37:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:48 DEBUG ----- output ----- 05:37:48 DEBUG { 05:37:48 DEBUG "status": "success", 05:37:48 DEBUG "data": { 05:37:48 DEBUG "resultType": "vector", 05:37:48 DEBUG "result": [ 05:37:48 DEBUG { 05:37:48 DEBUG "metric": { 05:37:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:48 DEBUG }, 05:37:48 DEBUG "value": [ 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG [...] 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:48 DEBUG }, 05:37:48 DEBUG "value": [ 05:37:48 DEBUG 1713760668, 05:37:48 DEBUG "1000" 05:37:48 DEBUG ] 05:37:48 DEBUG } 05:37:48 DEBUG ] 05:37:48 DEBUG } 05:37:48 DEBUG } 05:37:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760668" 05:37:48 INFO [http_cmd]: http status code OK 05:37:48 DEBUG --- status code --- 05:37:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:48 DEBUG --- http response --- 05:37:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:48 DEBUG ----- output ----- 05:37:48 DEBUG { 05:37:48 DEBUG "status": "success", 05:37:48 DEBUG "data": { 05:37:48 DEBUG "resultType": "vector", 05:37:48 DEBUG "result": [ 05:37:48 DEBUG { 05:37:48 DEBUG "metric": { 05:37:48 DEBUG "pod": "ds-cts-0" 05:37:48 DEBUG }, 05:37:48 DEBUG "value": [ 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG [...] 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG 05:37:48 DEBUG "pod": "ds-cts-2" 05:37:48 DEBUG }, 05:37:48 DEBUG "value": [ 05:37:48 DEBUG 1713760668, 05:37:48 DEBUG "35.06666666666666" 05:37:48 DEBUG ] 05:37:48 DEBUG } 05:37:48 DEBUG ] 05:37:48 DEBUG } 05:37:48 DEBUG } 05:37:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760669" 05:37:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760669" 05:37:49 INFO [http_cmd]: http status code OK 05:37:49 DEBUG --- status code --- 05:37:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:49 DEBUG --- http response --- 05:37:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:49 DEBUG ----- output ----- 05:37:49 DEBUG { 05:37:49 DEBUG "status": "success", 05:37:49 DEBUG "data": { 05:37:49 DEBUG "resultType": "vector", 05:37:49 DEBUG "result": [ 05:37:49 DEBUG { 05:37:49 DEBUG "metric": { 05:37:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:49 DEBUG }, 05:37:49 DEBUG "value": [ 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 INFO [http_cmd]: http status code OK 05:37:49 DEBUG [...] 05:37:49 DEBUG --- status code --- 05:37:49 DEBUG 05:37:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:49 DEBUG 05:37:49 DEBUG --- http response --- 05:37:49 DEBUG 05:37:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:49 DEBUG ----- output ----- 05:37:49 DEBUG }, 05:37:49 DEBUG { 05:37:49 DEBUG "value": [ 05:37:49 DEBUG "status": "success", 05:37:49 DEBUG 1713760669, 05:37:49 DEBUG "data": { 05:37:49 DEBUG "979.431" 05:37:49 DEBUG "resultType": "vector", 05:37:49 DEBUG ] 05:37:49 DEBUG "result": [ 05:37:49 DEBUG } 05:37:49 DEBUG { 05:37:49 DEBUG ] 05:37:49 DEBUG "metric": { 05:37:49 DEBUG } 05:37:49 DEBUG "pod": "ds-idrepo-0" 05:37:49 DEBUG } 05:37:49 DEBUG }, 05:37:49 DEBUG "value": [ 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG [...] 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG "pod": "ds-idrepo-2" 05:37:49 DEBUG }, 05:37:49 DEBUG "value": [ 05:37:49 DEBUG 1713760669, 05:37:49 DEBUG "0" 05:37:49 DEBUG ] 05:37:49 DEBUG } 05:37:49 DEBUG ] 05:37:49 DEBUG } 05:37:49 DEBUG } 05:37:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760669" 05:37:49 INFO [http_cmd]: http status code OK 05:37:49 DEBUG --- status code --- 05:37:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:49 DEBUG --- http response --- 05:37:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:49 DEBUG ----- output ----- 05:37:49 DEBUG { 05:37:49 DEBUG "status": "success", 05:37:49 DEBUG "data": { 05:37:49 DEBUG "resultType": "vector", 05:37:49 DEBUG "result": [ 05:37:49 DEBUG { 05:37:49 DEBUG "metric": { 05:37:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:37:49 DEBUG }, 05:37:49 DEBUG "value": [ 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG [...] 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG "pod": "ds-idrepo-0" 05:37:49 DEBUG }, 05:37:49 DEBUG "value": [ 05:37:49 DEBUG 1713760669, 05:37:49 DEBUG "5410.633691703665" 05:37:49 DEBUG ] 05:37:49 DEBUG } 05:37:49 DEBUG ] 05:37:49 DEBUG } 05:37:49 DEBUG } 05:37:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760669" 05:37:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760669" 05:37:49 INFO [http_cmd]: http status code OK 05:37:49 DEBUG --- status code --- 05:37:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:49 DEBUG --- http response --- 05:37:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:49 DEBUG ----- output ----- 05:37:49 DEBUG { 05:37:49 DEBUG "status": "success", 05:37:49 DEBUG "data": { 05:37:49 DEBUG "resultType": "vector", 05:37:49 DEBUG "result": [ 05:37:49 DEBUG { 05:37:49 DEBUG "metric": { 05:37:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:49 DEBUG }, 05:37:49 DEBUG "value": [ 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG [...] 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:49 DEBUG }, 05:37:49 DEBUG "value": [ 05:37:49 DEBUG 1713760669, 05:37:49 DEBUG "0" 05:37:49 DEBUG ] 05:37:49 DEBUG } 05:37:49 DEBUG ] 05:37:49 DEBUG } 05:37:49 DEBUG } 05:37:49 INFO [http_cmd]: http status code OK 05:37:49 DEBUG --- status code --- 05:37:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:49 DEBUG --- http response --- 05:37:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:49 DEBUG ----- output ----- 05:37:49 DEBUG { 05:37:49 DEBUG "status": "success", 05:37:49 DEBUG "data": { 05:37:49 DEBUG "resultType": "vector", 05:37:49 DEBUG "result": [ 05:37:49 DEBUG { 05:37:49 DEBUG "metric": { 05:37:49 DEBUG "instance": "10.142.0.107:9100" 05:37:49 DEBUG }, 05:37:49 DEBUG "value": [ 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG [...] 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG 05:37:49 DEBUG "instance": "10.142.0.43:9100" 05:37:49 DEBUG }, 05:37:49 DEBUG "value": [ 05:37:49 DEBUG 1713760669, 05:37:49 DEBUG "0.5064333333334445" 05:37:49 DEBUG ] 05:37:49 DEBUG } 05:37:49 DEBUG ] 05:37:49 DEBUG } 05:37:49 DEBUG } 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760670" 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760670" 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG { 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG "data": { 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "result": [ 05:37:50 DEBUG { 05:37:50 DEBUG "metric": { 05:37:50 DEBUG "instance": "10.142.0.107:9100" 05:37:50 DEBUG }, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG [...] 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG "instance": "10.142.0.43:9100" 05:37:50 DEBUG }, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG "5019862.466666667" 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG { 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG "data": { 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "result": [ 05:37:50 DEBUG { 05:37:50 DEBUG "metric": { 05:37:50 DEBUG "instance": "10.142.0.107:9100" 05:37:50 DEBUG }, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG [...] 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG "instance": "10.142.0.43:9100" 05:37:50 DEBUG }, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG "0.07000000000000457" 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760670" 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG { 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG "data": { 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "result": [ 05:37:50 DEBUG { 05:37:50 DEBUG "metric": { 05:37:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:37:50 DEBUG }, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG [...] 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:37:50 DEBUG }, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG "5429551104" 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760670" 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG { 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG "data": { 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "result": [ 05:37:50 DEBUG { 05:37:50 DEBUG "metric": { 05:37:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:37:50 DEBUG }, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG [...] 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:37:50 DEBUG }, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG "1.0340432429890813" 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760670" 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760670" 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760670" 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760670" 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760670" 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760670" 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760670" 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760670" 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760670" 05:37:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760670" 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 DEBUG --- status code --- 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG --- status code --- 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 DEBUG --- status code --- 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG --- http response --- 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG --- http response --- 05:37:50 INFO [http_cmd]: http status code OK 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:50 DEBUG --- status code --- 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG { 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG { 05:37:50 DEBUG { 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG { 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG { 05:37:50 DEBUG { 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG { 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG "data": { 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG { 05:37:50 DEBUG --- http response --- 05:37:50 DEBUG "data": { 05:37:50 DEBUG "data": { 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG "data": { 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "data": { 05:37:50 DEBUG "data": { 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "data": { 05:37:50 DEBUG { 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "result": [ 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "data": { 05:37:50 DEBUG "result": [ 05:37:50 DEBUG ----- output ----- 05:37:50 DEBUG "result": [ 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG "result": [ 05:37:50 DEBUG { 05:37:50 DEBUG "result": [ 05:37:50 DEBUG "result": [ 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG { 05:37:50 DEBUG { 05:37:50 DEBUG { 05:37:50 DEBUG "result": [ 05:37:50 DEBUG "data": { 05:37:50 DEBUG { 05:37:50 DEBUG "metric": { 05:37:50 DEBUG { 05:37:50 DEBUG { 05:37:50 DEBUG "result": [ 05:37:50 DEBUG "metric": { 05:37:50 DEBUG "status": "success", 05:37:50 DEBUG "metric": { 05:37:50 DEBUG { 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG "metric": { 05:37:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:37:50 DEBUG "metric": { 05:37:50 DEBUG "metric": { 05:37:50 DEBUG { 05:37:50 DEBUG "pod": "ds-idrepo-0" 05:37:50 DEBUG "data": { 05:37:50 DEBUG "pod": "ds-cts-0" 05:37:50 DEBUG "metric": { 05:37:50 DEBUG "result": [ 05:37:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:50 DEBUG }, 05:37:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:37:50 DEBUG "metric": { 05:37:50 DEBUG }, 05:37:50 DEBUG "resultType": "vector", 05:37:50 DEBUG }, 05:37:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:50 DEBUG { 05:37:50 DEBUG }, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG }, 05:37:50 DEBUG }, 05:37:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:50 DEBUG "value": [ 05:37:50 DEBUG "result": [ 05:37:50 DEBUG "value": [ 05:37:50 DEBUG }, 05:37:50 DEBUG "metric": { 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 05:37:50 DEBUG "value": [ 05:37:50 DEBUG "value": [ 05:37:50 DEBUG }, 05:37:50 DEBUG 05:37:50 DEBUG { 05:37:50 DEBUG 05:37:50 DEBUG "value": [ 05:37:50 DEBUG "pod": "ds-idrepo-0" 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 05:37:50 DEBUG "metric": { 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG }, 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG "instance": "10.142.0.107:9100" 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG "value": [ 05:37:50 DEBUG [...] 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG [...] 05:37:50 DEBUG }, 05:37:50 DEBUG [...] 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG [...] 05:37:50 DEBUG [...] 05:37:50 DEBUG [...] 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 05:37:50 DEBUG [...] 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG [...] 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG [...] 05:37:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG "pod": "ds-cts-2" 05:37:50 DEBUG 05:37:50 DEBUG "pod": "ds-cts-2" 05:37:50 DEBUG 05:37:50 DEBUG 05:37:50 DEBUG }, 05:37:50 DEBUG "pod": "ds-cts-0" 05:37:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:50 DEBUG "pod": "ds-cts-0" 05:37:50 DEBUG 05:37:50 DEBUG }, 05:37:50 DEBUG [...] 05:37:50 DEBUG }, 05:37:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:50 DEBUG 05:37:50 DEBUG "value": [ 05:37:50 DEBUG }, 05:37:50 DEBUG }, 05:37:50 DEBUG }, 05:37:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 05:37:50 DEBUG "value": [ 05:37:50 DEBUG }, 05:37:50 DEBUG 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG "value": [ 05:37:50 DEBUG "value": [ 05:37:50 DEBUG }, 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG "0" 05:37:50 DEBUG "value": [ 05:37:50 DEBUG "0" 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG "0" 05:37:50 DEBUG 05:37:50 DEBUG "pod": "ds-idrepo-2" 05:37:50 DEBUG ] 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG ] 05:37:50 DEBUG "18.37555408388645" 05:37:50 DEBUG "0" 05:37:50 DEBUG "80034884.13749605" 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG ] 05:37:50 DEBUG "instance": "10.142.0.43:9100" 05:37:50 DEBUG }, 05:37:50 DEBUG } 05:37:50 DEBUG "0" 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG ] 05:37:50 DEBUG ] 05:37:50 DEBUG "0" 05:37:50 DEBUG } 05:37:50 DEBUG }, 05:37:50 DEBUG "value": [ 05:37:50 DEBUG ] 05:37:50 DEBUG ] 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG ] 05:37:50 DEBUG "value": [ 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG ] 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG 1713760670, 05:37:50 DEBUG "0" 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG "5014498.466666667" 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG ] 05:37:50 DEBUG ] 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:37:50 DEBUG } 05:38:07 INFO 05:38:07 INFO [loop_until]: kubectl --namespace=xlou top pods 05:38:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:38:07 INFO [loop_until]: OK (rc = 0) 05:38:07 DEBUG --- stdout --- 05:38:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 956m 8791Mi am-767f6b67b5-qcqz2 972m 8839Mi am-767f6b67b5-vbltz 974m 8168Mi ds-cts-0 18453m 12510Mi ds-cts-1 17506m 12546Mi ds-cts-2 18377m 12556Mi ds-idrepo-0 14m 4010Mi ds-idrepo-1 12m 4002Mi ds-idrepo-2 13m 3985Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1000m 5411Mi 05:38:07 DEBUG --- stderr --- 05:38:07 DEBUG 05:38:15 INFO 05:38:15 INFO [loop_until]: kubectl --namespace=xlou top node 05:38:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:38:15 INFO [loop_until]: OK (rc = 0) 05:38:15 DEBUG --- stdout --- 05:38:15 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6240Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 8108Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 994m 3% 11526Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3353Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18096m 60% 15770Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5207Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1048m 3% 11737Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 75m 0% 3122Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1140m 3% 10403Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 7485Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17384m 58% 17717Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 88m 0% 8048Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18697m 62% 15256Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 15373Mi 26% 05:38:15 DEBUG --- stderr --- 05:38:15 DEBUG 05:38:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760727" 05:38:47 ERROR [http_cmd]: ERROR 05:38:47 ERROR --- status code --- 05:38:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:38:47 ERROR --- http response --- 05:38:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:38:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:38:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760727" 05:38:47 INFO [http_cmd]: http status code OK 05:38:47 DEBUG --- status code --- 05:38:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:47 DEBUG --- http response --- 05:38:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760727" 05:38:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760727" 05:38:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760727" 05:38:47 INFO [http_cmd]: http status code OK 05:38:47 DEBUG --- status code --- 05:38:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:47 DEBUG --- http response --- 05:38:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:38:47 DEBUG ----- output ----- 05:38:47 DEBUG { 05:38:47 DEBUG "status": "success", 05:38:47 DEBUG "data": { 05:38:47 DEBUG "resultType": "vector", 05:38:47 DEBUG "result": [ 05:38:47 DEBUG { 05:38:47 DEBUG "metric": { 05:38:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:38:47 DEBUG }, 05:38:47 DEBUG "value": [ 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG [...] 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:47 DEBUG }, 05:38:47 DEBUG "value": [ 05:38:47 DEBUG 1713760727, 05:38:47 DEBUG "0" 05:38:47 DEBUG ] 05:38:47 DEBUG } 05:38:47 DEBUG ] 05:38:47 DEBUG } 05:38:47 DEBUG } 05:38:47 INFO [http_cmd]: http status code OK 05:38:47 DEBUG --- status code --- 05:38:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:47 DEBUG --- http response --- 05:38:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:47 DEBUG ----- output ----- 05:38:47 DEBUG { 05:38:47 DEBUG "status": "success", 05:38:47 DEBUG "data": { 05:38:47 DEBUG "resultType": "vector", 05:38:47 DEBUG "result": [ 05:38:47 DEBUG { 05:38:47 DEBUG "metric": { 05:38:47 DEBUG "pod": "ds-idrepo-0" 05:38:47 DEBUG }, 05:38:47 DEBUG "value": [ 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG [...] 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG "pod": "ds-idrepo-2" 05:38:47 INFO [http_cmd]: http status code OK 05:38:47 DEBUG }, 05:38:47 DEBUG --- status code --- 05:38:47 DEBUG "value": [ 05:38:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:47 DEBUG 1713760727, 05:38:47 DEBUG --- http response --- 05:38:47 DEBUG "0" 05:38:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:47 DEBUG ] 05:38:47 DEBUG ----- output ----- 05:38:47 DEBUG } 05:38:47 DEBUG { 05:38:47 DEBUG ] 05:38:47 DEBUG "status": "success", 05:38:47 DEBUG } 05:38:47 DEBUG "data": { 05:38:47 DEBUG } 05:38:47 DEBUG "resultType": "vector", 05:38:47 DEBUG "result": [ 05:38:47 DEBUG { 05:38:47 DEBUG "metric": { 05:38:47 DEBUG "pod": "ds-cts-0" 05:38:47 DEBUG }, 05:38:47 DEBUG "value": [ 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG [...] 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG 05:38:47 DEBUG "pod": "ds-cts-2" 05:38:47 DEBUG }, 05:38:47 DEBUG "value": [ 05:38:47 DEBUG 1713760727, 05:38:47 DEBUG "86.9159" 05:38:47 DEBUG ] 05:38:47 DEBUG } 05:38:47 DEBUG ] 05:38:47 DEBUG } 05:38:47 DEBUG } 05:38:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760728" 05:38:48 INFO [http_cmd]: http status code OK 05:38:48 DEBUG --- status code --- 05:38:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:48 DEBUG --- http response --- 05:38:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:48 DEBUG ----- output ----- 05:38:48 DEBUG { 05:38:48 DEBUG "status": "success", 05:38:48 DEBUG "data": { 05:38:48 DEBUG "resultType": "vector", 05:38:48 DEBUG "result": [ 05:38:48 DEBUG { 05:38:48 DEBUG "metric": { 05:38:48 DEBUG "instance": "10.142.0.107:9100" 05:38:48 DEBUG }, 05:38:48 DEBUG "value": [ 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG [...] 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG "instance": "10.142.0.43:9100" 05:38:48 DEBUG }, 05:38:48 DEBUG "value": [ 05:38:48 DEBUG 1713760728, 05:38:48 DEBUG "5827403932672" 05:38:48 DEBUG ] 05:38:48 DEBUG } 05:38:48 DEBUG ] 05:38:48 DEBUG } 05:38:48 DEBUG } 05:38:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760728" 05:38:48 INFO [http_cmd]: http status code OK 05:38:48 DEBUG --- status code --- 05:38:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:48 DEBUG --- http response --- 05:38:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:48 DEBUG ----- output ----- 05:38:48 DEBUG { 05:38:48 DEBUG "status": "success", 05:38:48 DEBUG "data": { 05:38:48 DEBUG "resultType": "vector", 05:38:48 DEBUG "result": [ 05:38:48 DEBUG { 05:38:48 DEBUG "metric": { 05:38:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:48 DEBUG }, 05:38:48 DEBUG "value": [ 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG [...] 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:48 DEBUG }, 05:38:48 DEBUG "value": [ 05:38:48 DEBUG 1713760728, 05:38:48 DEBUG "1000" 05:38:48 DEBUG ] 05:38:48 DEBUG } 05:38:48 DEBUG ] 05:38:48 DEBUG } 05:38:48 DEBUG } 05:38:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760728" 05:38:48 INFO [http_cmd]: http status code OK 05:38:48 DEBUG --- status code --- 05:38:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:48 DEBUG --- http response --- 05:38:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:48 DEBUG ----- output ----- 05:38:48 DEBUG { 05:38:48 DEBUG "status": "success", 05:38:48 DEBUG "data": { 05:38:48 DEBUG "resultType": "vector", 05:38:48 DEBUG "result": [ 05:38:48 DEBUG { 05:38:48 DEBUG "metric": { 05:38:48 DEBUG "pod": "ds-cts-0" 05:38:48 DEBUG }, 05:38:48 DEBUG "value": [ 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG [...] 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG 05:38:48 DEBUG "pod": "ds-cts-2" 05:38:48 DEBUG }, 05:38:48 DEBUG "value": [ 05:38:48 DEBUG 1713760728, 05:38:48 DEBUG "38.43333333333333" 05:38:48 DEBUG ] 05:38:48 DEBUG } 05:38:48 DEBUG ] 05:38:48 DEBUG } 05:38:48 DEBUG } 05:38:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760729" 05:38:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760729" 05:38:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760729" 05:38:49 INFO [http_cmd]: http status code OK 05:38:49 INFO [http_cmd]: http status code OK 05:38:49 DEBUG --- status code --- 05:38:49 DEBUG --- status code --- 05:38:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:49 DEBUG --- http response --- 05:38:49 DEBUG --- http response --- 05:38:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:49 DEBUG ----- output ----- 05:38:49 DEBUG ----- output ----- 05:38:49 DEBUG { 05:38:49 DEBUG { 05:38:49 DEBUG "status": "success", 05:38:49 DEBUG "data": { 05:38:49 DEBUG "resultType": "vector", 05:38:49 DEBUG "result": [ 05:38:49 DEBUG { 05:38:49 DEBUG "metric": { 05:38:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:49 DEBUG }, 05:38:49 DEBUG "value": [ 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG [...] 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:49 DEBUG }, 05:38:49 DEBUG "value": [ 05:38:49 DEBUG 1713760729, 05:38:49 DEBUG "981.8405" 05:38:49 DEBUG ] 05:38:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760729" 05:38:49 DEBUG } 05:38:49 DEBUG "status": "success", 05:38:49 DEBUG ] 05:38:49 DEBUG "data": { 05:38:49 DEBUG } 05:38:49 DEBUG "resultType": "vector", 05:38:49 DEBUG } 05:38:49 DEBUG "result": [ 05:38:49 DEBUG { 05:38:49 DEBUG "metric": { 05:38:49 DEBUG "pod": "ds-idrepo-0" 05:38:49 DEBUG }, 05:38:49 DEBUG "value": [ 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG [...] 05:38:49 DEBUG 05:38:49 INFO [http_cmd]: http status code OK 05:38:49 DEBUG 05:38:49 DEBUG --- status code --- 05:38:49 DEBUG 05:38:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:49 DEBUG "pod": "ds-idrepo-2" 05:38:49 DEBUG --- http response --- 05:38:49 DEBUG }, 05:38:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:49 DEBUG "value": [ 05:38:49 DEBUG ----- output ----- 05:38:49 DEBUG { 05:38:49 DEBUG 1713760729, 05:38:49 DEBUG "status": "success", 05:38:49 DEBUG "0" 05:38:49 DEBUG "data": { 05:38:49 DEBUG ] 05:38:49 DEBUG "resultType": "vector", 05:38:49 DEBUG } 05:38:49 DEBUG "result": [ 05:38:49 DEBUG ] 05:38:49 DEBUG { 05:38:49 DEBUG } 05:38:49 DEBUG "metric": { 05:38:49 DEBUG } 05:38:49 DEBUG "instance": "10.142.0.107:9100" 05:38:49 DEBUG }, 05:38:49 DEBUG "value": [ 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG [...] 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG "instance": "10.142.0.43:9100" 05:38:49 DEBUG }, 05:38:49 DEBUG "value": [ 05:38:49 DEBUG 1713760729, 05:38:49 DEBUG "0.5202999999999235" 05:38:49 DEBUG ] 05:38:49 DEBUG } 05:38:49 DEBUG ] 05:38:49 DEBUG } 05:38:49 DEBUG } 05:38:49 INFO [http_cmd]: http status code OK 05:38:49 DEBUG --- status code --- 05:38:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:49 DEBUG --- http response --- 05:38:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:49 DEBUG ----- output ----- 05:38:49 DEBUG { 05:38:49 DEBUG "status": "success", 05:38:49 DEBUG "data": { 05:38:49 DEBUG "resultType": "vector", 05:38:49 DEBUG "result": [ 05:38:49 DEBUG { 05:38:49 DEBUG "metric": { 05:38:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:38:49 DEBUG }, 05:38:49 DEBUG "value": [ 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG [...] 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:38:49 DEBUG }, 05:38:49 DEBUG "value": [ 05:38:49 DEBUG 1713760729, 05:38:49 DEBUG "0" 05:38:49 DEBUG ] 05:38:49 DEBUG } 05:38:49 DEBUG ] 05:38:49 DEBUG } 05:38:49 DEBUG } 05:38:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760729" 05:38:49 INFO [http_cmd]: http status code OK 05:38:49 DEBUG --- status code --- 05:38:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:49 DEBUG --- http response --- 05:38:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:49 DEBUG ----- output ----- 05:38:49 DEBUG { 05:38:49 DEBUG "status": "success", 05:38:49 DEBUG "data": { 05:38:49 DEBUG "resultType": "vector", 05:38:49 DEBUG "result": [ 05:38:49 DEBUG { 05:38:49 DEBUG "metric": { 05:38:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:49 DEBUG }, 05:38:49 DEBUG "value": [ 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG [...] 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG 05:38:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:49 DEBUG }, 05:38:49 DEBUG "value": [ 05:38:49 DEBUG 1713760729, 05:38:49 DEBUG "0" 05:38:49 DEBUG ] 05:38:49 DEBUG } 05:38:49 DEBUG ] 05:38:49 DEBUG } 05:38:49 DEBUG } 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760730" 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760730" 05:38:50 INFO [http_cmd]: http status code OK 05:38:50 DEBUG --- status code --- 05:38:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:50 DEBUG --- http response --- 05:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:50 DEBUG ----- output ----- 05:38:50 DEBUG { 05:38:50 DEBUG "status": "success", 05:38:50 DEBUG "data": { 05:38:50 DEBUG "resultType": "vector", 05:38:50 DEBUG "result": [ 05:38:50 DEBUG { 05:38:50 DEBUG "metric": { 05:38:50 DEBUG "instance": "10.142.0.107:9100" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG [...] 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG "instance": "10.142.0.43:9100" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 1713760730, 05:38:50 DEBUG "4969205.633333333" 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG } 05:38:50 INFO [http_cmd]: http status code OK 05:38:50 DEBUG --- status code --- 05:38:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:50 DEBUG --- http response --- 05:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:50 DEBUG ----- output ----- 05:38:50 DEBUG { 05:38:50 DEBUG "status": "success", 05:38:50 DEBUG "data": { 05:38:50 DEBUG "resultType": "vector", 05:38:50 DEBUG "result": [ 05:38:50 DEBUG { 05:38:50 DEBUG "metric": { 05:38:50 DEBUG "instance": "10.142.0.107:9100" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG [...] 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG "instance": "10.142.0.43:9100" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 1713760730, 05:38:50 DEBUG "0.07433333333332068" 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG } 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760730" 05:38:50 INFO [http_cmd]: http status code OK 05:38:50 DEBUG --- status code --- 05:38:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:50 DEBUG --- http response --- 05:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:50 DEBUG ----- output ----- 05:38:50 DEBUG { 05:38:50 DEBUG "status": "success", 05:38:50 DEBUG "data": { 05:38:50 DEBUG "resultType": "vector", 05:38:50 DEBUG "result": [ 05:38:50 DEBUG { 05:38:50 DEBUG "metric": { 05:38:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG [...] 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 1713760730, 05:38:50 DEBUG "6042042368" 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG } 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760730" 05:38:50 INFO [http_cmd]: http status code OK 05:38:50 DEBUG --- status code --- 05:38:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:50 DEBUG --- http response --- 05:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:50 DEBUG ----- output ----- 05:38:50 DEBUG { 05:38:50 DEBUG "status": "success", 05:38:50 DEBUG "data": { 05:38:50 DEBUG "resultType": "vector", 05:38:50 DEBUG "result": [ 05:38:50 DEBUG { 05:38:50 DEBUG "metric": { 05:38:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG [...] 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 1713760730, 05:38:50 DEBUG "1.0307941315083682" 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG } 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760730" 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760730" 05:38:50 INFO [http_cmd]: http status code OK 05:38:50 DEBUG --- status code --- 05:38:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:50 DEBUG --- http response --- 05:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:50 DEBUG ----- output ----- 05:38:50 DEBUG { 05:38:50 DEBUG "status": "success", 05:38:50 DEBUG "data": { 05:38:50 DEBUG "resultType": "vector", 05:38:50 DEBUG "result": [ 05:38:50 DEBUG { 05:38:50 DEBUG "metric": { 05:38:50 DEBUG "pod": "ds-idrepo-0" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG [...] 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG "pod": "ds-cts-2" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 1713760730, 05:38:50 DEBUG "0" 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG } 05:38:50 INFO [http_cmd]: http status code OK 05:38:50 DEBUG --- status code --- 05:38:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:50 DEBUG --- http response --- 05:38:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:38:50 DEBUG ----- output ----- 05:38:50 DEBUG { 05:38:50 DEBUG "status": "success", 05:38:50 DEBUG "data": { 05:38:50 DEBUG "resultType": "vector", 05:38:50 DEBUG "result": [ 05:38:50 DEBUG { 05:38:50 DEBUG "metric": { 05:38:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG [...] 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG 05:38:50 DEBUG "pod": "ds-idrepo-0" 05:38:50 DEBUG }, 05:38:50 DEBUG "value": [ 05:38:50 DEBUG 1713760730, 05:38:50 DEBUG "0" 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG ] 05:38:50 DEBUG } 05:38:50 DEBUG } 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760730" 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760730" 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760730" 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760730" 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760730" 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760730" 05:38:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760730" 05:38:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760731" 05:38:51 INFO [http_cmd]: http status code OK 05:38:51 DEBUG --- status code --- 05:38:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:51 DEBUG --- http response --- 05:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:51 DEBUG ----- output ----- 05:38:51 DEBUG { 05:38:51 DEBUG "status": "success", 05:38:51 INFO [http_cmd]: http status code OK 05:38:51 INFO [http_cmd]: http status code OK 05:38:51 DEBUG "data": { 05:38:51 DEBUG --- status code --- 05:38:51 DEBUG --- status code --- 05:38:51 DEBUG "resultType": "vector", 05:38:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:51 INFO [http_cmd]: http status code OK 05:38:51 DEBUG "result": [ 05:38:51 INFO [http_cmd]: http status code OK 05:38:51 DEBUG --- http response --- 05:38:51 DEBUG --- http response --- 05:38:51 DEBUG --- status code --- 05:38:51 DEBUG { 05:38:51 DEBUG --- status code --- 05:38:51 INFO [http_cmd]: http status code OK 05:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:51 INFO [http_cmd]: http status code OK 05:38:51 INFO [http_cmd]: http status code OK 05:38:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:51 DEBUG "metric": { 05:38:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:51 DEBUG --- status code --- 05:38:51 DEBUG ----- output ----- 05:38:51 DEBUG { 05:38:51 DEBUG --- status code --- 05:38:51 DEBUG --- status code --- 05:38:51 DEBUG ----- output ----- 05:38:51 DEBUG "pod": "ds-cts-0" 05:38:51 DEBUG --- http response --- 05:38:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:51 DEBUG "status": "success", 05:38:51 DEBUG --- http response --- 05:38:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:51 DEBUG { 05:38:51 DEBUG }, 05:38:51 DEBUG --- http response --- 05:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:51 DEBUG "data": { 05:38:51 DEBUG --- http response --- 05:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:51 DEBUG --- http response --- 05:38:51 DEBUG "status": "success", 05:38:51 DEBUG "value": [ 05:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:51 DEBUG ----- output ----- 05:38:51 DEBUG "resultType": "vector", 05:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:51 DEBUG ----- output ----- 05:38:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:38:51 DEBUG "data": { 05:38:51 DEBUG 05:38:51 DEBUG ----- output ----- 05:38:51 DEBUG { 05:38:51 DEBUG "result": [ 05:38:51 DEBUG ----- output ----- 05:38:51 DEBUG { 05:38:51 DEBUG ----- output ----- 05:38:51 DEBUG "resultType": "vector", 05:38:51 DEBUG "result": [ 05:38:51 DEBUG { 05:38:51 DEBUG "status": "success", 05:38:51 DEBUG { 05:38:51 DEBUG { 05:38:51 DEBUG "status": "success", 05:38:51 DEBUG { 05:38:51 DEBUG 05:38:51 DEBUG { 05:38:51 DEBUG "status": "success", 05:38:51 DEBUG "data": { 05:38:51 DEBUG "metric": { 05:38:51 DEBUG "status": "success", 05:38:51 DEBUG "data": { 05:38:51 DEBUG "status": "success", 05:38:51 DEBUG 05:38:51 DEBUG "metric": { 05:38:51 DEBUG "data": { 05:38:51 DEBUG "resultType": "vector", 05:38:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:51 DEBUG "data": { 05:38:51 DEBUG "resultType": "vector", 05:38:51 DEBUG "data": { 05:38:51 DEBUG [...] 05:38:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:51 DEBUG "resultType": "vector", 05:38:51 DEBUG "result": [ 05:38:51 DEBUG }, 05:38:51 DEBUG "value": [ 05:38:51 DEBUG "result": [ 05:38:51 DEBUG "resultType": "vector", 05:38:51 DEBUG 05:38:51 DEBUG }, 05:38:51 DEBUG "result": [ 05:38:51 DEBUG { 05:38:51 DEBUG "resultType": "vector", 05:38:51 DEBUG 05:38:51 DEBUG { 05:38:51 DEBUG "result": [ 05:38:51 DEBUG 05:38:51 DEBUG "value": [ 05:38:51 DEBUG { 05:38:51 DEBUG "metric": { 05:38:51 DEBUG "result": [ 05:38:51 DEBUG 05:38:51 DEBUG "metric": { 05:38:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG "metric": { 05:38:51 DEBUG "instance": "10.142.0.107:9100" 05:38:51 DEBUG { 05:38:51 DEBUG 05:38:51 DEBUG }, 05:38:51 DEBUG { 05:38:51 DEBUG "pod": "ds-cts-2" 05:38:51 DEBUG 05:38:51 DEBUG "pod": "ds-idrepo-0" 05:38:51 DEBUG }, 05:38:51 DEBUG "metric": { 05:38:51 DEBUG [...] 05:38:51 DEBUG "value": [ 05:38:51 DEBUG "metric": { 05:38:51 DEBUG }, 05:38:51 DEBUG 05:38:51 DEBUG }, 05:38:51 DEBUG "value": [ 05:38:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:38:51 DEBUG "value": [ 05:38:51 DEBUG [...] 05:38:51 DEBUG "value": [ 05:38:51 DEBUG 05:38:51 DEBUG }, 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG }, 05:38:51 DEBUG "value": [ 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG "value": [ 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG 1713760730, 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:51 DEBUG [...] 05:38:51 DEBUG 05:38:51 DEBUG "0" 05:38:51 DEBUG 05:38:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:51 DEBUG [...] 05:38:51 DEBUG 05:38:51 DEBUG }, 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG ] 05:38:51 DEBUG 05:38:51 DEBUG }, 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG "value": [ 05:38:51 DEBUG 05:38:51 DEBUG [...] 05:38:51 DEBUG } 05:38:51 DEBUG [...] 05:38:51 DEBUG "value": [ 05:38:51 DEBUG 05:38:51 DEBUG [...] 05:38:51 DEBUG 1713760730, 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG ] 05:38:51 DEBUG 05:38:51 DEBUG 1713760730, 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG 05:38:51 DEBUG "0" 05:38:51 DEBUG 05:38:51 DEBUG } 05:38:51 DEBUG 05:38:51 DEBUG "0" 05:38:51 DEBUG "instance": "10.142.0.43:9100" 05:38:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:51 DEBUG 05:38:51 DEBUG ] 05:38:51 DEBUG 05:38:51 DEBUG } 05:38:51 DEBUG 05:38:51 DEBUG ] 05:38:51 DEBUG }, 05:38:51 DEBUG }, 05:38:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:51 DEBUG } 05:38:51 DEBUG "pod": "ds-idrepo-0" 05:38:51 DEBUG "pod": "ds-idrepo-2" 05:38:51 DEBUG } 05:38:51 DEBUG "value": [ 05:38:51 DEBUG "value": [ 05:38:51 DEBUG }, 05:38:51 DEBUG "value": [ 05:38:51 DEBUG }, 05:38:51 DEBUG }, 05:38:51 DEBUG ] 05:38:51 DEBUG 1713760730, 05:38:51 DEBUG 1713760730, 05:38:51 DEBUG 1713760731, 05:38:51 DEBUG "value": [ 05:38:51 DEBUG "value": [ 05:38:51 DEBUG } 05:38:51 DEBUG "4963102" 05:38:51 DEBUG "0" 05:38:51 DEBUG "0" 05:38:51 DEBUG 1713760730, 05:38:51 DEBUG ] 05:38:51 DEBUG 1713760730, 05:38:51 DEBUG } 05:38:51 DEBUG ] 05:38:51 DEBUG ] 05:38:51 DEBUG ] 05:38:51 DEBUG "0.007003531763354129" 05:38:51 DEBUG } 05:38:51 DEBUG "0" 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG ] 05:38:51 DEBUG } 05:38:51 DEBUG ] 05:38:51 DEBUG ] 05:38:51 DEBUG ] 05:38:51 DEBUG ] 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG ] 05:38:51 DEBUG ] 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG } 05:38:51 DEBUG } 05:39:07 INFO 05:39:07 INFO [loop_until]: kubectl --namespace=xlou top pods 05:39:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:39:07 INFO [loop_until]: OK (rc = 0) 05:39:07 DEBUG --- stdout --- 05:39:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 984m 8791Mi am-767f6b67b5-qcqz2 951m 8840Mi am-767f6b67b5-vbltz 1016m 8167Mi ds-cts-0 17324m 12511Mi ds-cts-1 17209m 12548Mi ds-cts-2 18328m 12553Mi ds-idrepo-0 14m 4009Mi ds-idrepo-1 12m 4001Mi ds-idrepo-2 12m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 8m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 997m 5978Mi 05:39:07 DEBUG --- stderr --- 05:39:07 DEBUG 05:39:15 INFO 05:39:15 INFO [loop_until]: kubectl --namespace=xlou top node 05:39:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:39:15 INFO [loop_until]: OK (rc = 0) 05:39:15 DEBUG --- stdout --- 05:39:15 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6230Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 8114Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1028m 3% 11522Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3345Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 17796m 59% 15764Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5216Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1046m 3% 11734Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 76m 0% 3129Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1112m 3% 10407Mi 9% gke-xlou-cdm-ds-0f601751-j488 64m 0% 7485Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17409m 58% 17726Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 90m 0% 8044Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18638m 62% 15275Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 15965Mi 27% 05:39:15 DEBUG --- stderr --- 05:39:15 DEBUG 05:39:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760787" 05:39:47 ERROR [http_cmd]: ERROR 05:39:47 ERROR --- status code --- 05:39:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:39:47 ERROR --- http response --- 05:39:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:39:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:39:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760787" 05:39:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760787" 05:39:47 INFO [http_cmd]: http status code OK 05:39:47 DEBUG --- status code --- 05:39:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:47 DEBUG --- http response --- 05:39:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:47 INFO [http_cmd]: http status code OK 05:39:47 DEBUG --- status code --- 05:39:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:47 DEBUG --- http response --- 05:39:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:39:47 DEBUG ----- output ----- 05:39:47 DEBUG { 05:39:47 DEBUG "status": "success", 05:39:47 DEBUG "data": { 05:39:47 DEBUG "resultType": "vector", 05:39:47 DEBUG "result": [ 05:39:47 DEBUG { 05:39:47 DEBUG "metric": { 05:39:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:39:47 DEBUG }, 05:39:47 DEBUG "value": [ 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG [...] 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:47 DEBUG }, 05:39:47 DEBUG "value": [ 05:39:47 DEBUG 1713760787, 05:39:47 DEBUG "24085.70097810899" 05:39:47 DEBUG ] 05:39:47 DEBUG } 05:39:47 DEBUG ] 05:39:47 DEBUG } 05:39:47 DEBUG } 05:39:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760787" 05:39:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760787" 05:39:47 INFO [http_cmd]: http status code OK 05:39:47 INFO [http_cmd]: http status code OK 05:39:47 DEBUG --- status code --- 05:39:47 DEBUG --- status code --- 05:39:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:47 DEBUG --- http response --- 05:39:47 DEBUG --- http response --- 05:39:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:47 DEBUG ----- output ----- 05:39:47 DEBUG { 05:39:47 DEBUG "status": "success", 05:39:47 DEBUG "data": { 05:39:47 DEBUG "resultType": "vector", 05:39:47 DEBUG "result": [ 05:39:47 DEBUG { 05:39:47 DEBUG "metric": { 05:39:47 DEBUG "pod": "ds-idrepo-0" 05:39:47 DEBUG }, 05:39:47 DEBUG "value": [ 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG [...] 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG "pod": "ds-idrepo-2" 05:39:47 DEBUG }, 05:39:47 DEBUG "value": [ 05:39:47 DEBUG 1713760787, 05:39:47 DEBUG "0" 05:39:47 DEBUG ] 05:39:47 DEBUG } 05:39:47 DEBUG ] 05:39:47 DEBUG } 05:39:47 DEBUG } 05:39:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:47 DEBUG ----- output ----- 05:39:47 DEBUG { 05:39:47 DEBUG "status": "success", 05:39:47 DEBUG "data": { 05:39:47 DEBUG "resultType": "vector", 05:39:47 DEBUG "result": [ 05:39:47 DEBUG { 05:39:47 DEBUG "metric": { 05:39:47 DEBUG "pod": "ds-cts-0" 05:39:47 DEBUG }, 05:39:47 DEBUG "value": [ 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG [...] 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG 05:39:47 DEBUG "pod": "ds-cts-2" 05:39:47 DEBUG }, 05:39:47 DEBUG "value": [ 05:39:47 DEBUG 1713760787, 05:39:47 DEBUG "84.83046666666667" 05:39:47 DEBUG ] 05:39:47 DEBUG } 05:39:47 DEBUG ] 05:39:47 DEBUG } 05:39:47 DEBUG } 05:39:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760788" 05:39:48 INFO [http_cmd]: http status code OK 05:39:48 DEBUG --- status code --- 05:39:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:48 DEBUG --- http response --- 05:39:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:48 DEBUG ----- output ----- 05:39:48 DEBUG { 05:39:48 DEBUG "status": "success", 05:39:48 DEBUG "data": { 05:39:48 DEBUG "resultType": "vector", 05:39:48 DEBUG "result": [ 05:39:48 DEBUG { 05:39:48 DEBUG "metric": { 05:39:48 DEBUG "instance": "10.142.0.107:9100" 05:39:48 DEBUG }, 05:39:48 DEBUG "value": [ 05:39:48 DEBUG 05:39:48 DEBUG 05:39:48 DEBUG 05:39:48 DEBUG [...] 05:39:48 DEBUG 05:39:48 DEBUG 05:39:48 DEBUG 05:39:48 DEBUG "instance": "10.142.0.43:9100" 05:39:48 DEBUG }, 05:39:48 DEBUG "value": [ 05:39:48 DEBUG 1713760788, 05:39:48 DEBUG "5831543426048" 05:39:48 DEBUG ] 05:39:48 DEBUG } 05:39:48 DEBUG ] 05:39:48 DEBUG } 05:39:48 DEBUG } 05:39:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760788" 05:39:48 INFO [http_cmd]: http status code OK 05:39:48 DEBUG --- status code --- 05:39:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:48 DEBUG --- http response --- 05:39:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:48 DEBUG ----- output ----- 05:39:48 DEBUG { 05:39:48 DEBUG "status": "success", 05:39:48 DEBUG "data": { 05:39:48 DEBUG "resultType": "vector", 05:39:48 DEBUG "result": [ 05:39:48 DEBUG { 05:39:48 DEBUG "metric": { 05:39:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:48 DEBUG }, 05:39:48 DEBUG "value": [ 05:39:48 DEBUG 05:39:48 DEBUG 05:39:48 DEBUG 05:39:48 DEBUG [...] 05:39:48 DEBUG 05:39:48 DEBUG 05:39:48 DEBUG 05:39:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:48 DEBUG }, 05:39:48 DEBUG "value": [ 05:39:48 DEBUG 1713760788, 05:39:48 DEBUG "1000" 05:39:48 DEBUG ] 05:39:48 DEBUG } 05:39:48 DEBUG ] 05:39:48 DEBUG } 05:39:48 DEBUG } 05:39:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760789" 05:39:49 INFO [http_cmd]: http status code OK 05:39:49 DEBUG --- status code --- 05:39:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:49 DEBUG --- http response --- 05:39:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:49 DEBUG ----- output ----- 05:39:49 DEBUG { 05:39:49 DEBUG "status": "success", 05:39:49 DEBUG "data": { 05:39:49 DEBUG "resultType": "vector", 05:39:49 DEBUG "result": [ 05:39:49 DEBUG { 05:39:49 DEBUG "metric": { 05:39:49 DEBUG "pod": "ds-cts-0" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG [...] 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG "pod": "ds-cts-2" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 1713760789, 05:39:49 DEBUG "38.4" 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG } 05:39:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760789" 05:39:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760789" 05:39:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760789" 05:39:49 INFO [http_cmd]: http status code OK 05:39:49 DEBUG --- status code --- 05:39:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:49 DEBUG --- http response --- 05:39:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:49 DEBUG ----- output ----- 05:39:49 DEBUG { 05:39:49 DEBUG "status": "success", 05:39:49 DEBUG "data": { 05:39:49 DEBUG "resultType": "vector", 05:39:49 DEBUG "result": [ 05:39:49 DEBUG { 05:39:49 DEBUG "metric": { 05:39:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG [...] 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 1713760789, 05:39:49 DEBUG "984.1785" 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG } 05:39:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760789" 05:39:49 INFO [http_cmd]: http status code OK 05:39:49 DEBUG --- status code --- 05:39:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:49 DEBUG --- http response --- 05:39:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:49 DEBUG ----- output ----- 05:39:49 DEBUG { 05:39:49 DEBUG "status": "success", 05:39:49 DEBUG "data": { 05:39:49 DEBUG "resultType": "vector", 05:39:49 DEBUG "result": [ 05:39:49 DEBUG { 05:39:49 DEBUG "metric": { 05:39:49 DEBUG "pod": "ds-idrepo-0" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG [...] 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG "pod": "ds-idrepo-2" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 1713760789, 05:39:49 DEBUG "0" 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG } 05:39:49 INFO [http_cmd]: http status code OK 05:39:49 DEBUG --- status code --- 05:39:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:49 DEBUG --- http response --- 05:39:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:49 DEBUG ----- output ----- 05:39:49 DEBUG { 05:39:49 DEBUG "status": "success", 05:39:49 DEBUG "data": { 05:39:49 DEBUG "resultType": "vector", 05:39:49 DEBUG "result": [ 05:39:49 DEBUG { 05:39:49 DEBUG "metric": { 05:39:49 DEBUG "instance": "10.142.0.107:9100" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG [...] 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG "instance": "10.142.0.43:9100" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 1713760789, 05:39:49 DEBUG "0.47049999999989794" 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG } 05:39:49 INFO [http_cmd]: http status code OK 05:39:49 DEBUG --- status code --- 05:39:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:49 DEBUG --- http response --- 05:39:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:39:49 DEBUG ----- output ----- 05:39:49 DEBUG { 05:39:49 DEBUG "status": "success", 05:39:49 DEBUG "data": { 05:39:49 DEBUG "resultType": "vector", 05:39:49 DEBUG "result": [ 05:39:49 DEBUG { 05:39:49 DEBUG "metric": { 05:39:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG [...] 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 1713760789, 05:39:49 DEBUG "476.9445738239404" 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG } 05:39:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760789" 05:39:49 INFO [http_cmd]: http status code OK 05:39:49 DEBUG --- status code --- 05:39:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:49 DEBUG --- http response --- 05:39:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:49 DEBUG ----- output ----- 05:39:49 DEBUG { 05:39:49 DEBUG "status": "success", 05:39:49 DEBUG "data": { 05:39:49 DEBUG "resultType": "vector", 05:39:49 DEBUG "result": [ 05:39:49 DEBUG { 05:39:49 DEBUG "metric": { 05:39:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG [...] 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG 05:39:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:49 DEBUG }, 05:39:49 DEBUG "value": [ 05:39:49 DEBUG 1713760789, 05:39:49 DEBUG "0" 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG ] 05:39:49 DEBUG } 05:39:49 DEBUG } 05:39:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760790" 05:39:50 INFO [http_cmd]: http status code OK 05:39:50 DEBUG --- status code --- 05:39:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:50 DEBUG --- http response --- 05:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:50 DEBUG ----- output ----- 05:39:50 DEBUG { 05:39:50 DEBUG "status": "success", 05:39:50 DEBUG "data": { 05:39:50 DEBUG "resultType": "vector", 05:39:50 DEBUG "result": [ 05:39:50 DEBUG { 05:39:50 DEBUG "metric": { 05:39:50 DEBUG "instance": "10.142.0.107:9100" 05:39:50 DEBUG }, 05:39:50 DEBUG "value": [ 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG [...] 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG "instance": "10.142.0.43:9100" 05:39:50 DEBUG }, 05:39:50 DEBUG "value": [ 05:39:50 DEBUG 1713760790, 05:39:50 DEBUG "0.16966666666665822" 05:39:50 DEBUG ] 05:39:50 DEBUG } 05:39:50 DEBUG ] 05:39:50 DEBUG } 05:39:50 DEBUG } 05:39:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760790" 05:39:50 INFO [http_cmd]: http status code OK 05:39:50 DEBUG --- status code --- 05:39:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:50 DEBUG --- http response --- 05:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:50 DEBUG ----- output ----- 05:39:50 DEBUG { 05:39:50 DEBUG "status": "success", 05:39:50 DEBUG "data": { 05:39:50 DEBUG "resultType": "vector", 05:39:50 DEBUG "result": [ 05:39:50 DEBUG { 05:39:50 DEBUG "metric": { 05:39:50 DEBUG "instance": "10.142.0.107:9100" 05:39:50 DEBUG }, 05:39:50 DEBUG "value": [ 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG [...] 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG "instance": "10.142.0.43:9100" 05:39:50 DEBUG }, 05:39:50 DEBUG "value": [ 05:39:50 DEBUG 1713760790, 05:39:50 DEBUG "5521804.533333333" 05:39:50 DEBUG ] 05:39:50 DEBUG } 05:39:50 DEBUG ] 05:39:50 DEBUG } 05:39:50 DEBUG } 05:39:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760790" 05:39:50 INFO [http_cmd]: http status code OK 05:39:50 DEBUG --- status code --- 05:39:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:50 DEBUG --- http response --- 05:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:50 DEBUG ----- output ----- 05:39:50 DEBUG { 05:39:50 DEBUG "status": "success", 05:39:50 DEBUG "data": { 05:39:50 DEBUG "resultType": "vector", 05:39:50 DEBUG "result": [ 05:39:50 DEBUG { 05:39:50 DEBUG "metric": { 05:39:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:39:50 DEBUG }, 05:39:50 DEBUG "value": [ 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG [...] 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:39:50 DEBUG }, 05:39:50 DEBUG "value": [ 05:39:50 DEBUG 1713760790, 05:39:50 DEBUG "6724669440" 05:39:50 DEBUG ] 05:39:50 DEBUG } 05:39:50 DEBUG ] 05:39:50 DEBUG } 05:39:50 DEBUG } 05:39:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760790" 05:39:50 INFO [http_cmd]: http status code OK 05:39:50 DEBUG --- status code --- 05:39:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:50 DEBUG --- http response --- 05:39:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:50 DEBUG ----- output ----- 05:39:50 DEBUG { 05:39:50 DEBUG "status": "success", 05:39:50 DEBUG "data": { 05:39:50 DEBUG "resultType": "vector", 05:39:50 DEBUG "result": [ 05:39:50 DEBUG { 05:39:50 DEBUG "metric": { 05:39:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:39:50 DEBUG }, 05:39:50 DEBUG "value": [ 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG [...] 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG 05:39:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:39:50 DEBUG }, 05:39:50 DEBUG "value": [ 05:39:50 DEBUG 1713760790, 05:39:50 DEBUG "1.0356075159164073" 05:39:50 DEBUG ] 05:39:50 DEBUG } 05:39:50 DEBUG ] 05:39:50 DEBUG } 05:39:50 DEBUG } 05:39:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760791" 05:39:51 INFO [http_cmd]: http status code OK 05:39:51 DEBUG --- status code --- 05:39:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:51 DEBUG --- http response --- 05:39:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760791" 05:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:51 DEBUG ----- output ----- 05:39:51 DEBUG { 05:39:51 DEBUG "status": "success", 05:39:51 DEBUG "data": { 05:39:51 DEBUG "resultType": "vector", 05:39:51 DEBUG "result": [ 05:39:51 DEBUG { 05:39:51 DEBUG "metric": { 05:39:51 DEBUG "pod": "ds-idrepo-0" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG [...] 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG "pod": "ds-cts-2" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 1713760791, 05:39:51 DEBUG "0" 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760791" 05:39:51 INFO [http_cmd]: http status code OK 05:39:51 DEBUG --- status code --- 05:39:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:51 DEBUG --- http response --- 05:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:51 DEBUG ----- output ----- 05:39:51 DEBUG { 05:39:51 DEBUG "status": "success", 05:39:51 DEBUG "data": { 05:39:51 DEBUG "resultType": "vector", 05:39:51 DEBUG "result": [ 05:39:51 DEBUG { 05:39:51 DEBUG "metric": { 05:39:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG [...] 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 1713760791, 05:39:51 DEBUG "0" 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760791" 05:39:51 INFO [http_cmd]: http status code OK 05:39:51 DEBUG --- status code --- 05:39:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:51 DEBUG --- http response --- 05:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:51 DEBUG ----- output ----- 05:39:51 DEBUG { 05:39:51 DEBUG "status": "success", 05:39:51 DEBUG "data": { 05:39:51 DEBUG "resultType": "vector", 05:39:51 DEBUG "result": [ 05:39:51 DEBUG { 05:39:51 DEBUG "metric": { 05:39:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG [...] 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 1713760791, 05:39:51 DEBUG "0" 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760791" 05:39:51 INFO [http_cmd]: http status code OK 05:39:51 DEBUG --- status code --- 05:39:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:51 DEBUG --- http response --- 05:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:51 DEBUG ----- output ----- 05:39:51 DEBUG { 05:39:51 DEBUG "status": "success", 05:39:51 DEBUG "data": { 05:39:51 DEBUG "resultType": "vector", 05:39:51 DEBUG "result": [ 05:39:51 DEBUG { 05:39:51 DEBUG "metric": { 05:39:51 DEBUG "pod": "ds-cts-0" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG [...] 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG "pod": "ds-cts-2" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 1713760791, 05:39:51 DEBUG "0" 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760791" 05:39:51 INFO [http_cmd]: http status code OK 05:39:51 DEBUG --- status code --- 05:39:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:51 DEBUG --- http response --- 05:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:51 DEBUG ----- output ----- 05:39:51 DEBUG { 05:39:51 DEBUG "status": "success", 05:39:51 DEBUG "data": { 05:39:51 DEBUG "resultType": "vector", 05:39:51 DEBUG "result": [ 05:39:51 DEBUG { 05:39:51 DEBUG "metric": { 05:39:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG [...] 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 1713760791, 05:39:51 DEBUG "0" 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 INFO [http_cmd]: http status code OK 05:39:51 DEBUG --- status code --- 05:39:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:51 DEBUG --- http response --- 05:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:51 DEBUG ----- output ----- 05:39:51 DEBUG { 05:39:51 DEBUG "status": "success", 05:39:51 DEBUG "data": { 05:39:51 DEBUG "resultType": "vector", 05:39:51 DEBUG "result": [ 05:39:51 DEBUG { 05:39:51 DEBUG "metric": { 05:39:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG [...] 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 1713760791, 05:39:51 DEBUG "0" 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760791" 05:39:51 INFO [http_cmd]: http status code OK 05:39:51 DEBUG --- status code --- 05:39:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:51 DEBUG --- http response --- 05:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:39:51 DEBUG ----- output ----- 05:39:51 DEBUG { 05:39:51 DEBUG "status": "success", 05:39:51 DEBUG "data": { 05:39:51 DEBUG "resultType": "vector", 05:39:51 DEBUG "result": [ 05:39:51 DEBUG { 05:39:51 DEBUG "metric": { 05:39:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG [...] 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 1713760791, 05:39:51 DEBUG "0" 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760791" 05:39:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760791" 05:39:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760791" 05:39:51 INFO [http_cmd]: http status code OK 05:39:51 DEBUG --- status code --- 05:39:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:51 DEBUG --- http response --- 05:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:51 DEBUG ----- output ----- 05:39:51 DEBUG { 05:39:51 DEBUG "status": "success", 05:39:51 DEBUG "data": { 05:39:51 DEBUG "resultType": "vector", 05:39:51 DEBUG "result": [ 05:39:51 DEBUG { 05:39:51 DEBUG "metric": { 05:39:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG [...] 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 1713760791, 05:39:51 DEBUG "0" 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 INFO [http_cmd]: http status code OK 05:39:51 INFO [http_cmd]: http status code OK 05:39:51 DEBUG --- status code --- 05:39:51 DEBUG --- status code --- 05:39:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:51 DEBUG --- http response --- 05:39:51 DEBUG --- http response --- 05:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:51 DEBUG ----- output ----- 05:39:51 DEBUG ----- output ----- 05:39:51 DEBUG { 05:39:51 DEBUG { 05:39:51 DEBUG "status": "success", 05:39:51 DEBUG "status": "success", 05:39:51 DEBUG "data": { 05:39:51 DEBUG "data": { 05:39:51 DEBUG "resultType": "vector", 05:39:51 DEBUG "resultType": "vector", 05:39:51 DEBUG "result": [ 05:39:51 DEBUG "result": [ 05:39:51 DEBUG { 05:39:51 DEBUG { 05:39:51 DEBUG "metric": { 05:39:51 DEBUG "metric": { 05:39:51 DEBUG "pod": "ds-idrepo-0" 05:39:51 DEBUG "instance": "10.142.0.107:9100" 05:39:51 DEBUG }, 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG [...] 05:39:51 DEBUG [...] 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG 05:39:51 DEBUG "pod": "ds-idrepo-2" 05:39:51 DEBUG "instance": "10.142.0.43:9100" 05:39:51 DEBUG }, 05:39:51 DEBUG }, 05:39:51 DEBUG "value": [ 05:39:51 DEBUG "value": [ 05:39:51 DEBUG 1713760791, 05:39:51 DEBUG 1713760791, 05:39:51 DEBUG "0" 05:39:51 DEBUG "5517577.966666667" 05:39:51 DEBUG ] 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 DEBUG ] 05:39:51 DEBUG ] 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 DEBUG } 05:39:51 DEBUG } 05:40:08 INFO 05:40:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:40:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:40:08 INFO [loop_until]: OK (rc = 0) 05:40:08 DEBUG --- stdout --- 05:40:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 1184m 8792Mi am-767f6b67b5-qcqz2 1092m 8840Mi am-767f6b67b5-vbltz 1040m 8166Mi ds-cts-0 18438m 12517Mi ds-cts-1 11729m 11392Mi ds-cts-2 18430m 12551Mi ds-idrepo-0 14m 4009Mi ds-idrepo-1 13m 4001Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1009m 6467Mi 05:40:08 DEBUG --- stderr --- 05:40:08 DEBUG 05:40:15 INFO 05:40:15 INFO [loop_until]: kubectl --namespace=xlou top node 05:40:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:40:15 INFO [loop_until]: OK (rc = 0) 05:40:15 DEBUG --- stdout --- 05:40:15 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6232Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 8106Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1181m 3% 11527Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3353Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18031m 60% 15763Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 136m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1120m 3% 11732Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 73m 0% 3132Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1164m 3% 10427Mi 9% gke-xlou-cdm-ds-0f601751-j488 68m 0% 7480Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 11186m 37% 16556Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 90m 0% 8046Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18437m 61% 15271Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 16552Mi 28% 05:40:15 DEBUG --- stderr --- 05:40:15 DEBUG 05:40:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760847" 05:40:47 ERROR [http_cmd]: ERROR 05:40:47 ERROR --- status code --- 05:40:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:40:47 ERROR --- http response --- 05:40:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:40:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:40:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760847" 05:40:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760847" 05:40:47 INFO [http_cmd]: http status code OK 05:40:47 DEBUG --- status code --- 05:40:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:47 DEBUG --- http response --- 05:40:47 INFO [http_cmd]: http status code OK 05:40:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:47 DEBUG --- status code --- 05:40:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:47 DEBUG --- http response --- 05:40:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:47 DEBUG ----- output ----- 05:40:47 DEBUG { 05:40:47 DEBUG "status": "success", 05:40:47 DEBUG "data": { 05:40:47 DEBUG "resultType": "vector", 05:40:47 DEBUG "result": [ 05:40:47 DEBUG { 05:40:47 DEBUG "metric": { 05:40:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:40:47 DEBUG }, 05:40:47 DEBUG "value": [ 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG [...] 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:47 DEBUG }, 05:40:47 DEBUG "value": [ 05:40:47 DEBUG 1713760847, 05:40:47 DEBUG "26956.041785503334" 05:40:47 DEBUG ] 05:40:47 DEBUG } 05:40:47 DEBUG ] 05:40:47 DEBUG } 05:40:47 DEBUG } 05:40:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760847" 05:40:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760847" 05:40:47 INFO [http_cmd]: http status code OK 05:40:47 INFO [http_cmd]: http status code OK 05:40:47 DEBUG --- status code --- 05:40:47 DEBUG --- status code --- 05:40:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:47 DEBUG --- http response --- 05:40:47 DEBUG --- http response --- 05:40:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:47 DEBUG ----- output ----- 05:40:47 DEBUG ----- output ----- 05:40:47 DEBUG { 05:40:47 DEBUG { 05:40:47 DEBUG "status": "success", 05:40:47 DEBUG "status": "success", 05:40:47 DEBUG "data": { 05:40:47 DEBUG "data": { 05:40:47 DEBUG "resultType": "vector", 05:40:47 DEBUG "resultType": "vector", 05:40:47 DEBUG "result": [ 05:40:47 DEBUG "result": [ 05:40:47 DEBUG { 05:40:47 DEBUG { 05:40:47 DEBUG "metric": { 05:40:47 DEBUG "metric": { 05:40:47 DEBUG "pod": "ds-idrepo-0" 05:40:47 DEBUG "pod": "ds-cts-0" 05:40:47 DEBUG }, 05:40:47 DEBUG }, 05:40:47 DEBUG "value": [ 05:40:47 DEBUG "value": [ 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG [...] 05:40:47 DEBUG [...] 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG 05:40:47 DEBUG "pod": "ds-idrepo-2" 05:40:47 DEBUG "pod": "ds-cts-2" 05:40:47 DEBUG }, 05:40:47 DEBUG }, 05:40:47 DEBUG "value": [ 05:40:47 DEBUG "value": [ 05:40:47 DEBUG 1713760847, 05:40:47 DEBUG 1713760847, 05:40:47 DEBUG "0" 05:40:47 DEBUG "82.63493333333332" 05:40:47 DEBUG ] 05:40:47 DEBUG ] 05:40:47 DEBUG } 05:40:47 DEBUG } 05:40:47 DEBUG ] 05:40:47 DEBUG ] 05:40:47 DEBUG } 05:40:47 DEBUG } 05:40:47 DEBUG } 05:40:47 DEBUG } 05:40:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760848" 05:40:48 INFO [http_cmd]: http status code OK 05:40:48 DEBUG --- status code --- 05:40:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:48 DEBUG --- http response --- 05:40:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:48 DEBUG ----- output ----- 05:40:48 DEBUG { 05:40:48 DEBUG "status": "success", 05:40:48 DEBUG "data": { 05:40:48 DEBUG "resultType": "vector", 05:40:48 DEBUG "result": [ 05:40:48 DEBUG { 05:40:48 DEBUG "metric": { 05:40:48 DEBUG "instance": "10.142.0.107:9100" 05:40:48 DEBUG }, 05:40:48 DEBUG "value": [ 05:40:48 DEBUG 05:40:48 DEBUG 05:40:48 DEBUG 05:40:48 DEBUG [...] 05:40:48 DEBUG 05:40:48 DEBUG 05:40:48 DEBUG 05:40:48 DEBUG "instance": "10.142.0.43:9100" 05:40:48 DEBUG }, 05:40:48 DEBUG "value": [ 05:40:48 DEBUG 1713760848, 05:40:48 DEBUG "5832686056448" 05:40:48 DEBUG ] 05:40:48 DEBUG } 05:40:48 DEBUG ] 05:40:48 DEBUG } 05:40:48 DEBUG } 05:40:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760849" 05:40:49 INFO [http_cmd]: http status code OK 05:40:49 DEBUG --- status code --- 05:40:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:49 DEBUG --- http response --- 05:40:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:49 DEBUG ----- output ----- 05:40:49 DEBUG { 05:40:49 DEBUG "status": "success", 05:40:49 DEBUG "data": { 05:40:49 DEBUG "resultType": "vector", 05:40:49 DEBUG "result": [ 05:40:49 DEBUG { 05:40:49 DEBUG "metric": { 05:40:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG [...] 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 1713760849, 05:40:49 DEBUG "833.3333333333334" 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG } 05:40:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760849" 05:40:49 INFO [http_cmd]: http status code OK 05:40:49 DEBUG --- status code --- 05:40:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:49 DEBUG --- http response --- 05:40:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:49 DEBUG ----- output ----- 05:40:49 DEBUG { 05:40:49 DEBUG "status": "success", 05:40:49 DEBUG "data": { 05:40:49 DEBUG "resultType": "vector", 05:40:49 DEBUG "result": [ 05:40:49 DEBUG { 05:40:49 DEBUG "metric": { 05:40:49 DEBUG "pod": "ds-cts-0" 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG [...] 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG "pod": "ds-cts-2" 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 1713760849, 05:40:49 DEBUG "39.6" 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG } 05:40:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760849" 05:40:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760849" 05:40:49 INFO [http_cmd]: http status code OK 05:40:49 DEBUG --- status code --- 05:40:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:49 DEBUG --- http response --- 05:40:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:49 DEBUG ----- output ----- 05:40:49 DEBUG { 05:40:49 DEBUG "status": "success", 05:40:49 DEBUG "data": { 05:40:49 DEBUG "resultType": "vector", 05:40:49 DEBUG "result": [ 05:40:49 DEBUG { 05:40:49 DEBUG "metric": { 05:40:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG [...] 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 1713760849, 05:40:49 DEBUG "986.4010000000001" 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG } 05:40:49 INFO [http_cmd]: http status code OK 05:40:49 DEBUG --- status code --- 05:40:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:49 DEBUG --- http response --- 05:40:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:49 DEBUG ----- output ----- 05:40:49 DEBUG { 05:40:49 DEBUG "status": "success", 05:40:49 DEBUG "data": { 05:40:49 DEBUG "resultType": "vector", 05:40:49 DEBUG "result": [ 05:40:49 DEBUG { 05:40:49 DEBUG "metric": { 05:40:49 DEBUG "pod": "ds-idrepo-0" 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG [...] 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG "pod": "ds-idrepo-2" 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 1713760849, 05:40:49 DEBUG "0" 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG } 05:40:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760849" 05:40:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760849" 05:40:49 INFO [http_cmd]: http status code OK 05:40:49 INFO [http_cmd]: http status code OK 05:40:49 DEBUG --- status code --- 05:40:49 DEBUG --- status code --- 05:40:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:49 DEBUG --- http response --- 05:40:49 DEBUG --- http response --- 05:40:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:40:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:49 DEBUG ----- output ----- 05:40:49 DEBUG ----- output ----- 05:40:49 DEBUG { 05:40:49 DEBUG { 05:40:49 DEBUG "status": "success", 05:40:49 DEBUG "status": "success", 05:40:49 DEBUG "data": { 05:40:49 DEBUG "data": { 05:40:49 DEBUG "resultType": "vector", 05:40:49 DEBUG "resultType": "vector", 05:40:49 DEBUG "result": [ 05:40:49 DEBUG "result": [ 05:40:49 DEBUG { 05:40:49 DEBUG { 05:40:49 DEBUG "metric": { 05:40:49 DEBUG "metric": { 05:40:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:40:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:49 DEBUG }, 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760849" 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG [...] 05:40:49 DEBUG [...] 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:49 DEBUG }, 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 1713760849, 05:40:49 DEBUG 1713760849, 05:40:49 DEBUG "476.14430287176884" 05:40:49 DEBUG "0" 05:40:49 DEBUG ] 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG } 05:40:49 DEBUG ] 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG } 05:40:49 DEBUG } 05:40:49 DEBUG } 05:40:49 INFO [http_cmd]: http status code OK 05:40:49 DEBUG --- status code --- 05:40:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:49 DEBUG --- http response --- 05:40:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:49 DEBUG ----- output ----- 05:40:49 DEBUG { 05:40:49 DEBUG "status": "success", 05:40:49 DEBUG "data": { 05:40:49 DEBUG "resultType": "vector", 05:40:49 DEBUG "result": [ 05:40:49 DEBUG { 05:40:49 DEBUG "metric": { 05:40:49 DEBUG "instance": "10.142.0.107:9100" 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG [...] 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG 05:40:49 DEBUG "instance": "10.142.0.43:9100" 05:40:49 DEBUG }, 05:40:49 DEBUG "value": [ 05:40:49 DEBUG 1713760849, 05:40:49 DEBUG "0.0011666666665481292" 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG ] 05:40:49 DEBUG } 05:40:49 DEBUG } 05:40:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760850" 05:40:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760850" 05:40:50 INFO [http_cmd]: http status code OK 05:40:50 DEBUG --- status code --- 05:40:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:50 DEBUG --- http response --- 05:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:50 INFO [http_cmd]: http status code OK 05:40:50 DEBUG ----- output ----- 05:40:50 DEBUG --- status code --- 05:40:50 DEBUG { 05:40:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:50 DEBUG "status": "success", 05:40:50 DEBUG --- http response --- 05:40:50 DEBUG "data": { 05:40:50 DEBUG "resultType": "vector", 05:40:50 DEBUG "result": [ 05:40:50 DEBUG { 05:40:50 DEBUG "metric": { 05:40:50 DEBUG "instance": "10.142.0.107:9100" 05:40:50 DEBUG }, 05:40:50 DEBUG "value": [ 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG [...] 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG "instance": "10.142.0.43:9100" 05:40:50 DEBUG }, 05:40:50 DEBUG "value": [ 05:40:50 DEBUG 1713760850, 05:40:50 DEBUG "0.0006666666666660604" 05:40:50 DEBUG ] 05:40:50 DEBUG } 05:40:50 DEBUG ] 05:40:50 DEBUG } 05:40:50 DEBUG } 05:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:50 DEBUG ----- output ----- 05:40:50 DEBUG { 05:40:50 DEBUG "status": "success", 05:40:50 DEBUG "data": { 05:40:50 DEBUG "resultType": "vector", 05:40:50 DEBUG "result": [ 05:40:50 DEBUG { 05:40:50 DEBUG "metric": { 05:40:50 DEBUG "instance": "10.142.0.107:9100" 05:40:50 DEBUG }, 05:40:50 DEBUG "value": [ 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG [...] 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG "instance": "10.142.0.43:9100" 05:40:50 DEBUG }, 05:40:50 DEBUG "value": [ 05:40:50 DEBUG 1713760850, 05:40:50 DEBUG "1707249.6333333333" 05:40:50 DEBUG ] 05:40:50 DEBUG } 05:40:50 DEBUG ] 05:40:50 DEBUG } 05:40:50 DEBUG } 05:40:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760850" 05:40:50 INFO [http_cmd]: http status code OK 05:40:50 DEBUG --- status code --- 05:40:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:50 DEBUG --- http response --- 05:40:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:50 DEBUG ----- output ----- 05:40:50 DEBUG { 05:40:50 DEBUG "status": "success", 05:40:50 DEBUG "data": { 05:40:50 DEBUG "resultType": "vector", 05:40:50 DEBUG "result": [ 05:40:50 DEBUG { 05:40:50 DEBUG "metric": { 05:40:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:40:50 DEBUG }, 05:40:50 DEBUG "value": [ 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG [...] 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG 05:40:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:40:50 DEBUG }, 05:40:50 DEBUG "value": [ 05:40:50 DEBUG 1713760850, 05:40:50 DEBUG "7320637440" 05:40:50 DEBUG ] 05:40:50 DEBUG } 05:40:50 DEBUG ] 05:40:50 DEBUG } 05:40:50 DEBUG } 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760851" 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "1.0281897868654004" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760851" 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "0" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760851" 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "pod": "ds-idrepo-0" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "pod": "ds-cts-2" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "0" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760851" 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760851" 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "0" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760851" 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760851" 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760851" 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "0" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "pod": "ds-cts-0" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "pod": "ds-cts-2" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "0" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760851" 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "0" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "instance": "10.142.0.107:9100" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "instance": "10.142.0.43:9100" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "1700946.6" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "0" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760851" 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "0.43208818626966483" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:40:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760851" 05:40:51 INFO [http_cmd]: http status code OK 05:40:51 DEBUG --- status code --- 05:40:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:51 DEBUG --- http response --- 05:40:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:51 DEBUG ----- output ----- 05:40:51 DEBUG { 05:40:51 DEBUG "status": "success", 05:40:51 DEBUG "data": { 05:40:51 DEBUG "resultType": "vector", 05:40:51 DEBUG "result": [ 05:40:51 DEBUG { 05:40:51 DEBUG "metric": { 05:40:51 DEBUG "pod": "ds-idrepo-0" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG [...] 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG 05:40:51 DEBUG "pod": "ds-idrepo-2" 05:40:51 DEBUG }, 05:40:51 DEBUG "value": [ 05:40:51 DEBUG 1713760851, 05:40:51 DEBUG "0" 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG ] 05:40:51 DEBUG } 05:40:51 DEBUG } 05:41:08 INFO 05:41:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:41:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:41:08 INFO [loop_until]: OK (rc = 0) 05:41:08 DEBUG --- stdout --- 05:41:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 762m 8792Mi am-767f6b67b5-qcqz2 764m 8840Mi am-767f6b67b5-vbltz 638m 8165Mi ds-cts-0 18442m 12510Mi ds-cts-1 236m 11392Mi ds-cts-2 18526m 12551Mi ds-idrepo-0 15m 4010Mi ds-idrepo-1 13m 4000Mi ds-idrepo-2 14m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 3m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1021m 7148Mi 05:41:08 DEBUG --- stderr --- 05:41:08 DEBUG 05:41:15 INFO 05:41:15 INFO [loop_until]: kubectl --namespace=xlou top node 05:41:15 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:41:15 INFO [loop_until]: OK (rc = 0) 05:41:15 DEBUG --- stdout --- 05:41:15 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6242Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 8108Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 877m 2% 11518Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18111m 60% 15772Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5202Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 837m 2% 11734Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 76m 0% 3129Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 690m 2% 10415Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 7460Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 303m 1% 16565Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 90m 0% 8048Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18719m 62% 15267Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1042m 6% 17136Mi 29% 05:41:15 DEBUG --- stderr --- 05:41:15 DEBUG 05:41:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760907" 05:41:47 ERROR [http_cmd]: ERROR 05:41:47 ERROR --- status code --- 05:41:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:41:47 ERROR --- http response --- 05:41:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:41:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:41:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760907" 05:41:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760907" 05:41:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760907" 05:41:47 INFO [http_cmd]: http status code OK 05:41:47 INFO [http_cmd]: http status code OK 05:41:47 DEBUG --- status code --- 05:41:47 DEBUG --- status code --- 05:41:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:47 DEBUG --- http response --- 05:41:47 DEBUG --- http response --- 05:41:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:41:47 DEBUG ----- output ----- 05:41:47 DEBUG { 05:41:47 DEBUG "status": "success", 05:41:47 DEBUG "data": { 05:41:47 DEBUG "resultType": "vector", 05:41:47 DEBUG "result": [ 05:41:47 DEBUG { 05:41:47 DEBUG "metric": { 05:41:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:41:47 DEBUG }, 05:41:47 DEBUG "value": [ 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG [...] 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG "pod": "ds-cts-2" 05:41:47 DEBUG }, 05:41:47 DEBUG "value": [ 05:41:47 DEBUG 1713760907, 05:41:47 DEBUG "377849134.6154695" 05:41:47 DEBUG ] 05:41:47 DEBUG } 05:41:47 DEBUG ] 05:41:47 DEBUG } 05:41:47 DEBUG } 05:41:47 INFO [http_cmd]: http status code OK 05:41:47 DEBUG --- status code --- 05:41:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:47 DEBUG --- http response --- 05:41:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:47 DEBUG ----- output ----- 05:41:47 DEBUG { 05:41:47 DEBUG "status": "success", 05:41:47 DEBUG "data": { 05:41:47 DEBUG "resultType": "vector", 05:41:47 DEBUG "result": [ 05:41:47 DEBUG { 05:41:47 DEBUG "metric": { 05:41:47 DEBUG "pod": "ds-idrepo-0" 05:41:47 DEBUG }, 05:41:47 DEBUG "value": [ 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG [...] 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG "pod": "ds-idrepo-2" 05:41:47 DEBUG }, 05:41:47 DEBUG "value": [ 05:41:47 DEBUG 1713760907, 05:41:47 DEBUG "0" 05:41:47 DEBUG ] 05:41:47 DEBUG } 05:41:47 DEBUG ] 05:41:47 DEBUG } 05:41:47 DEBUG } 05:41:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760907" 05:41:47 INFO [http_cmd]: http status code OK 05:41:47 DEBUG --- status code --- 05:41:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:47 DEBUG --- http response --- 05:41:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:47 DEBUG ----- output ----- 05:41:47 DEBUG { 05:41:47 DEBUG "status": "success", 05:41:47 DEBUG "data": { 05:41:47 DEBUG "resultType": "vector", 05:41:47 DEBUG "result": [ 05:41:47 DEBUG { 05:41:47 DEBUG "metric": { 05:41:47 DEBUG "pod": "ds-cts-0" 05:41:47 DEBUG }, 05:41:47 DEBUG "value": [ 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG [...] 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG 05:41:47 DEBUG "pod": "ds-cts-2" 05:41:47 DEBUG }, 05:41:47 DEBUG "value": [ 05:41:47 DEBUG 1713760907, 05:41:47 DEBUG "80.42606666666666" 05:41:47 DEBUG ] 05:41:47 DEBUG } 05:41:47 DEBUG ] 05:41:47 DEBUG } 05:41:47 DEBUG } 05:41:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760908" 05:41:48 INFO [http_cmd]: http status code OK 05:41:48 DEBUG --- status code --- 05:41:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:48 DEBUG --- http response --- 05:41:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:48 DEBUG ----- output ----- 05:41:48 DEBUG { 05:41:48 DEBUG "status": "success", 05:41:48 DEBUG "data": { 05:41:48 DEBUG "resultType": "vector", 05:41:48 DEBUG "result": [ 05:41:48 DEBUG { 05:41:48 DEBUG "metric": { 05:41:48 DEBUG "instance": "10.142.0.107:9100" 05:41:48 DEBUG }, 05:41:48 DEBUG "value": [ 05:41:48 DEBUG 05:41:48 DEBUG 05:41:48 DEBUG 05:41:48 DEBUG [...] 05:41:48 DEBUG 05:41:48 DEBUG 05:41:48 DEBUG 05:41:48 DEBUG "instance": "10.142.0.43:9100" 05:41:48 DEBUG }, 05:41:48 DEBUG "value": [ 05:41:48 DEBUG 1713760908, 05:41:48 DEBUG "5832687801344" 05:41:48 DEBUG ] 05:41:48 DEBUG } 05:41:48 DEBUG ] 05:41:48 DEBUG } 05:41:48 DEBUG } 05:41:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760909" 05:41:49 INFO [http_cmd]: http status code OK 05:41:49 DEBUG --- status code --- 05:41:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:49 DEBUG --- http response --- 05:41:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:49 DEBUG ----- output ----- 05:41:49 DEBUG { 05:41:49 DEBUG "status": "success", 05:41:49 DEBUG "data": { 05:41:49 DEBUG "resultType": "vector", 05:41:49 DEBUG "result": [ 05:41:49 DEBUG { 05:41:49 DEBUG "metric": { 05:41:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG [...] 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 1713760909, 05:41:49 DEBUG "0" 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG } 05:41:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760909" 05:41:49 INFO [http_cmd]: http status code OK 05:41:49 DEBUG --- status code --- 05:41:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:49 DEBUG --- http response --- 05:41:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:49 DEBUG ----- output ----- 05:41:49 DEBUG { 05:41:49 DEBUG "status": "success", 05:41:49 DEBUG "data": { 05:41:49 DEBUG "resultType": "vector", 05:41:49 DEBUG "result": [ 05:41:49 DEBUG { 05:41:49 DEBUG "metric": { 05:41:49 DEBUG "pod": "ds-cts-0" 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG [...] 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG "pod": "ds-cts-2" 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 1713760909, 05:41:49 DEBUG "35.666666666666664" 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG } 05:41:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760909" 05:41:49 INFO [http_cmd]: http status code OK 05:41:49 DEBUG --- status code --- 05:41:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:49 DEBUG --- http response --- 05:41:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:49 DEBUG ----- output ----- 05:41:49 DEBUG { 05:41:49 DEBUG "status": "success", 05:41:49 DEBUG "data": { 05:41:49 DEBUG "resultType": "vector", 05:41:49 DEBUG "result": [ 05:41:49 DEBUG { 05:41:49 DEBUG "metric": { 05:41:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG [...] 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 1713760909, 05:41:49 DEBUG "987.589" 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG } 05:41:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760909" 05:41:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760909" 05:41:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760909" 05:41:49 INFO [http_cmd]: http status code OK 05:41:49 DEBUG --- status code --- 05:41:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:49 DEBUG --- http response --- 05:41:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:49 DEBUG ----- output ----- 05:41:49 DEBUG { 05:41:49 DEBUG "status": "success", 05:41:49 DEBUG "data": { 05:41:49 DEBUG "resultType": "vector", 05:41:49 DEBUG "result": [ 05:41:49 DEBUG { 05:41:49 DEBUG "metric": { 05:41:49 DEBUG "pod": "ds-idrepo-0" 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG [...] 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG "pod": "ds-idrepo-2" 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 1713760909, 05:41:49 DEBUG "0" 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG } 05:41:49 INFO [http_cmd]: http status code OK 05:41:49 INFO [http_cmd]: http status code OK 05:41:49 DEBUG --- status code --- 05:41:49 DEBUG --- status code --- 05:41:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:49 DEBUG --- http response --- 05:41:49 DEBUG --- http response --- 05:41:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:41:49 DEBUG ----- output ----- 05:41:49 DEBUG ----- output ----- 05:41:49 DEBUG { 05:41:49 DEBUG { 05:41:49 DEBUG "status": "success", 05:41:49 DEBUG "status": "success", 05:41:49 DEBUG "data": { 05:41:49 DEBUG "data": { 05:41:49 DEBUG "resultType": "vector", 05:41:49 DEBUG "resultType": "vector", 05:41:49 DEBUG "result": [ 05:41:49 DEBUG "result": [ 05:41:49 DEBUG { 05:41:49 DEBUG { 05:41:49 DEBUG "metric": { 05:41:49 DEBUG "metric": { 05:41:49 DEBUG "instance": "10.142.0.107:9100" 05:41:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:41:49 DEBUG }, 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG [...] 05:41:49 DEBUG [...] 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG "instance": "10.142.0.43:9100" 05:41:49 DEBUG "pod": "ds-cts-2" 05:41:49 DEBUG }, 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 1713760909, 05:41:49 DEBUG 1713760909, 05:41:49 DEBUG "0.000733333333171989" 05:41:49 DEBUG "24950127.74679128" 05:41:49 DEBUG ] 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG } 05:41:49 DEBUG ] 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG } 05:41:49 DEBUG } 05:41:49 DEBUG } 05:41:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760909" 05:41:49 INFO [http_cmd]: http status code OK 05:41:49 DEBUG --- status code --- 05:41:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:49 DEBUG --- http response --- 05:41:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:49 DEBUG ----- output ----- 05:41:49 DEBUG { 05:41:49 DEBUG "status": "success", 05:41:49 DEBUG "data": { 05:41:49 DEBUG "resultType": "vector", 05:41:49 DEBUG "result": [ 05:41:49 DEBUG { 05:41:49 DEBUG "metric": { 05:41:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG [...] 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG 05:41:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:49 DEBUG }, 05:41:49 DEBUG "value": [ 05:41:49 DEBUG 1713760909, 05:41:49 DEBUG "0" 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG ] 05:41:49 DEBUG } 05:41:49 DEBUG } 05:41:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760910" 05:41:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760910" 05:41:50 INFO [http_cmd]: http status code OK 05:41:50 DEBUG --- status code --- 05:41:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:50 DEBUG --- http response --- 05:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:50 DEBUG ----- output ----- 05:41:50 DEBUG { 05:41:50 DEBUG "status": "success", 05:41:50 DEBUG "data": { 05:41:50 DEBUG "resultType": "vector", 05:41:50 DEBUG "result": [ 05:41:50 DEBUG { 05:41:50 DEBUG "metric": { 05:41:50 DEBUG "instance": "10.142.0.107:9100" 05:41:50 DEBUG }, 05:41:50 DEBUG "value": [ 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG [...] 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG "instance": "10.142.0.43:9100" 05:41:50 DEBUG }, 05:41:50 DEBUG "value": [ 05:41:50 DEBUG 1713760910, 05:41:50 DEBUG "1151992.1666666667" 05:41:50 DEBUG ] 05:41:50 DEBUG } 05:41:50 DEBUG ] 05:41:50 DEBUG } 05:41:50 DEBUG } 05:41:50 INFO [http_cmd]: http status code OK 05:41:50 DEBUG --- status code --- 05:41:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:50 DEBUG --- http response --- 05:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:50 DEBUG ----- output ----- 05:41:50 DEBUG { 05:41:50 DEBUG "status": "success", 05:41:50 DEBUG "data": { 05:41:50 DEBUG "resultType": "vector", 05:41:50 DEBUG "result": [ 05:41:50 DEBUG { 05:41:50 DEBUG "metric": { 05:41:50 DEBUG "instance": "10.142.0.107:9100" 05:41:50 DEBUG }, 05:41:50 DEBUG "value": [ 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG [...] 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG "instance": "10.142.0.43:9100" 05:41:50 DEBUG }, 05:41:50 DEBUG "value": [ 05:41:50 DEBUG 1713760910, 05:41:50 DEBUG "0.0003333333333330302" 05:41:50 DEBUG ] 05:41:50 DEBUG } 05:41:50 DEBUG ] 05:41:50 DEBUG } 05:41:50 DEBUG } 05:41:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760910" 05:41:50 INFO [http_cmd]: http status code OK 05:41:50 DEBUG --- status code --- 05:41:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:50 DEBUG --- http response --- 05:41:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:50 DEBUG ----- output ----- 05:41:50 DEBUG { 05:41:50 DEBUG "status": "success", 05:41:50 DEBUG "data": { 05:41:50 DEBUG "resultType": "vector", 05:41:50 DEBUG "result": [ 05:41:50 DEBUG { 05:41:50 DEBUG "metric": { 05:41:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:41:50 DEBUG }, 05:41:50 DEBUG "value": [ 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG [...] 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG 05:41:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:41:50 DEBUG }, 05:41:50 DEBUG "value": [ 05:41:50 DEBUG 1713760910, 05:41:50 DEBUG "7532957696" 05:41:50 DEBUG ] 05:41:50 DEBUG } 05:41:50 DEBUG ] 05:41:50 DEBUG } 05:41:50 DEBUG } 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760911" 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760911" 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG --- http response --- 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG { 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "data": { 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "result": [ 05:41:51 DEBUG { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG [...] 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG "pod": "ds-idrepo-0" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG "0" 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG --- http response --- 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG { 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "data": { 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "result": [ 05:41:51 DEBUG { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG [...] 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG "0.49871914389431177" 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760911" 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG --- http response --- 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG { 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "data": { 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "result": [ 05:41:51 DEBUG { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG [...] 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG "0" 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760911" 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG --- http response --- 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG { 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "data": { 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "result": [ 05:41:51 DEBUG { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG [...] 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG "0" 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760911" 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760911" 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG --- http response --- 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760911" 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG { 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "data": { 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "result": [ 05:41:51 DEBUG { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "pod": "ds-idrepo-0" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG [...] 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG "pod": "ds-cts-2" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG "0" 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760911" 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG --- http response --- 05:41:51 DEBUG --- http response --- 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG { 05:41:51 DEBUG { 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "data": { 05:41:51 DEBUG "data": { 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "result": [ 05:41:51 DEBUG "result": [ 05:41:51 DEBUG { 05:41:51 DEBUG { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "pod": "ds-idrepo-0" 05:41:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:51 DEBUG }, 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG [...] 05:41:51 DEBUG [...] 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG "pod": "ds-idrepo-2" 05:41:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:51 DEBUG }, 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG "0" 05:41:51 DEBUG "0" 05:41:51 DEBUG ] 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 DEBUG ] 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG --- http response --- 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760911" 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG { 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "data": { 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "result": [ 05:41:51 DEBUG { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG [...] 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG "0" 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760911" 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG --- http response --- 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG { 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "data": { 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "result": [ 05:41:51 DEBUG { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "pod": "ds-cts-0" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG [...] 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG "pod": "ds-cts-2" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG "0" 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG --- http response --- 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG { 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "data": { 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "result": [ 05:41:51 DEBUG { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "instance": "10.142.0.107:9100" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG [...] 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG "instance": "10.142.0.43:9100" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG "1145701.9" 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:41:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760911" 05:41:51 INFO [http_cmd]: http status code OK 05:41:51 DEBUG --- status code --- 05:41:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:51 DEBUG --- http response --- 05:41:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:41:51 DEBUG ----- output ----- 05:41:51 DEBUG { 05:41:51 DEBUG "status": "success", 05:41:51 DEBUG "data": { 05:41:51 DEBUG "resultType": "vector", 05:41:51 DEBUG "result": [ 05:41:51 DEBUG { 05:41:51 DEBUG "metric": { 05:41:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG [...] 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG 05:41:51 DEBUG "pod": "ds-idrepo-0" 05:41:51 DEBUG }, 05:41:51 DEBUG "value": [ 05:41:51 DEBUG 1713760911, 05:41:51 DEBUG "0.013320771335820826" 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG ] 05:41:51 DEBUG } 05:41:51 DEBUG } 05:42:08 INFO 05:42:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:42:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:42:08 INFO [loop_until]: OK (rc = 0) 05:42:08 DEBUG --- stdout --- 05:42:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 789m 8792Mi am-767f6b67b5-qcqz2 676m 8840Mi am-767f6b67b5-vbltz 652m 8165Mi ds-cts-0 18388m 12502Mi ds-cts-1 219m 11385Mi ds-cts-2 18369m 12551Mi ds-idrepo-0 14m 4009Mi ds-idrepo-1 13m 4001Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-68c5d5b884-p97w5 959m 7027Mi 05:42:08 DEBUG --- stderr --- 05:42:08 DEBUG 05:42:16 INFO 05:42:16 INFO [loop_until]: kubectl --namespace=xlou top node 05:42:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:42:16 INFO [loop_until]: OK (rc = 0) 05:42:16 DEBUG --- stdout --- 05:42:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6236Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 8113Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 837m 2% 11529Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3355Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18494m 61% 15747Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 5193Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 783m 2% 11719Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 75m 0% 3139Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 730m 2% 10424Mi 9% gke-xlou-cdm-ds-0f601751-j488 70m 0% 7468Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 293m 0% 16562Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 94m 0% 8035Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18616m 62% 15275Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1015m 6% 17106Mi 29% 05:42:16 DEBUG --- stderr --- 05:42:16 DEBUG 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713760967" 05:42:47 ERROR [http_cmd]: ERROR 05:42:47 ERROR --- status code --- 05:42:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:42:47 ERROR --- http response --- 05:42:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:42:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760967" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760967" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760967" 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "data": { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "result": [ 05:42:47 DEBUG { 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "pod": "ds-idrepo-0" 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG [...] 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "pod": "ds-idrepo-2" 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 1713760967, 05:42:47 DEBUG "0" 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "data": { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "result": [ 05:42:47 DEBUG { 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 1713760967, 05:42:47 DEBUG "20633619.481826425" 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760967" 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "data": { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "result": [ 05:42:47 DEBUG { 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "pod": "ds-cts-0" 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "pod": "ds-cts-2" 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 1713760967, 05:42:47 DEBUG "78.27653333333333" 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760968" 05:42:48 INFO [http_cmd]: http status code OK 05:42:48 DEBUG --- status code --- 05:42:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:48 DEBUG --- http response --- 05:42:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:48 DEBUG ----- output ----- 05:42:48 DEBUG { 05:42:48 DEBUG "status": "success", 05:42:48 DEBUG "data": { 05:42:48 DEBUG "resultType": "vector", 05:42:48 DEBUG "result": [ 05:42:48 DEBUG { 05:42:48 DEBUG "metric": { 05:42:48 DEBUG "instance": "10.142.0.107:9100" 05:42:48 DEBUG }, 05:42:48 DEBUG "value": [ 05:42:48 DEBUG 05:42:48 DEBUG 05:42:48 DEBUG 05:42:48 DEBUG [...] 05:42:48 DEBUG 05:42:48 DEBUG 05:42:48 DEBUG 05:42:48 DEBUG "instance": "10.142.0.43:9100" 05:42:48 DEBUG }, 05:42:48 DEBUG "value": [ 05:42:48 DEBUG 1713760968, 05:42:48 DEBUG "5832690846720" 05:42:48 DEBUG ] 05:42:48 DEBUG } 05:42:48 DEBUG ] 05:42:48 DEBUG } 05:42:48 DEBUG } 05:42:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760969" 05:42:49 INFO [http_cmd]: http status code OK 05:42:49 DEBUG --- status code --- 05:42:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:49 DEBUG --- http response --- 05:42:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:49 DEBUG ----- output ----- 05:42:49 DEBUG { 05:42:49 DEBUG "status": "success", 05:42:49 DEBUG "data": { 05:42:49 DEBUG "resultType": "vector", 05:42:49 DEBUG "result": [ 05:42:49 DEBUG { 05:42:49 DEBUG "metric": { 05:42:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG [...] 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 1713760969, 05:42:49 DEBUG "833.3333333333334" 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG } 05:42:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760969" 05:42:49 INFO [http_cmd]: http status code OK 05:42:49 DEBUG --- status code --- 05:42:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:49 DEBUG --- http response --- 05:42:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:49 DEBUG ----- output ----- 05:42:49 DEBUG { 05:42:49 DEBUG "status": "success", 05:42:49 DEBUG "data": { 05:42:49 DEBUG "resultType": "vector", 05:42:49 DEBUG "result": [ 05:42:49 DEBUG { 05:42:49 DEBUG "metric": { 05:42:49 DEBUG "pod": "ds-cts-0" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG [...] 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG "pod": "ds-cts-2" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 1713760969, 05:42:49 DEBUG "23.833333333333332" 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG } 05:42:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713760969" 05:42:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760969" 05:42:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760969" 05:42:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760969" 05:42:49 INFO [http_cmd]: http status code OK 05:42:49 INFO [http_cmd]: http status code OK 05:42:49 DEBUG --- status code --- 05:42:49 DEBUG --- status code --- 05:42:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:49 DEBUG --- http response --- 05:42:49 DEBUG --- http response --- 05:42:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:49 DEBUG ----- output ----- 05:42:49 DEBUG { 05:42:49 DEBUG "status": "success", 05:42:49 DEBUG "data": { 05:42:49 DEBUG "resultType": "vector", 05:42:49 DEBUG "result": [ 05:42:49 DEBUG { 05:42:49 DEBUG "metric": { 05:42:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG [...] 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 1713760969, 05:42:49 DEBUG "989.0070000000001" 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG } 05:42:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:49 DEBUG ----- output ----- 05:42:49 DEBUG { 05:42:49 DEBUG "status": "success", 05:42:49 DEBUG "data": { 05:42:49 DEBUG "resultType": "vector", 05:42:49 DEBUG "result": [ 05:42:49 DEBUG { 05:42:49 DEBUG "metric": { 05:42:49 DEBUG "instance": "10.142.0.107:9100" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG [...] 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG "instance": "10.142.0.43:9100" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 1713760969, 05:42:49 DEBUG "0.0004333333334746688" 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG } 05:42:49 INFO [http_cmd]: http status code OK 05:42:49 DEBUG --- status code --- 05:42:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:49 DEBUG --- http response --- 05:42:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:49 DEBUG ----- output ----- 05:42:49 DEBUG { 05:42:49 DEBUG "status": "success", 05:42:49 DEBUG "data": { 05:42:49 DEBUG "resultType": "vector", 05:42:49 DEBUG "result": [ 05:42:49 DEBUG { 05:42:49 DEBUG "metric": { 05:42:49 DEBUG "pod": "ds-idrepo-0" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG [...] 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG "pod": "ds-idrepo-2" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 1713760969, 05:42:49 DEBUG "0" 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG } 05:42:49 INFO [http_cmd]: http status code OK 05:42:49 DEBUG --- status code --- 05:42:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:49 DEBUG --- http response --- 05:42:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:49 DEBUG ----- output ----- 05:42:49 DEBUG { 05:42:49 DEBUG "status": "success", 05:42:49 DEBUG "data": { 05:42:49 DEBUG "resultType": "vector", 05:42:49 DEBUG "result": [ 05:42:49 DEBUG { 05:42:49 DEBUG "metric": { 05:42:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG [...] 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 1713760969, 05:42:49 DEBUG "0" 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG } 05:42:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760969" 05:42:49 INFO [http_cmd]: http status code OK 05:42:49 DEBUG --- status code --- 05:42:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:49 DEBUG --- http response --- 05:42:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:42:49 DEBUG ----- output ----- 05:42:49 DEBUG { 05:42:49 DEBUG "status": "success", 05:42:49 DEBUG "data": { 05:42:49 DEBUG "resultType": "vector", 05:42:49 DEBUG "result": [ 05:42:49 DEBUG { 05:42:49 DEBUG "metric": { 05:42:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG [...] 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG 05:42:49 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:42:49 DEBUG }, 05:42:49 DEBUG "value": [ 05:42:49 DEBUG 1713760969, 05:42:49 DEBUG "0" 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG ] 05:42:49 DEBUG } 05:42:49 DEBUG } 05:42:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760970" 05:42:50 INFO [http_cmd]: http status code OK 05:42:50 DEBUG --- status code --- 05:42:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:50 DEBUG --- http response --- 05:42:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:50 DEBUG ----- output ----- 05:42:50 DEBUG { 05:42:50 DEBUG "status": "success", 05:42:50 DEBUG "data": { 05:42:50 DEBUG "resultType": "vector", 05:42:50 DEBUG "result": [ 05:42:50 DEBUG { 05:42:50 DEBUG "metric": { 05:42:50 DEBUG "instance": "10.142.0.107:9100" 05:42:50 DEBUG }, 05:42:50 DEBUG "value": [ 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG [...] 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG "instance": "10.142.0.43:9100" 05:42:50 DEBUG }, 05:42:50 DEBUG "value": [ 05:42:50 DEBUG 1713760970, 05:42:50 DEBUG "1387630.1666666667" 05:42:50 DEBUG ] 05:42:50 DEBUG } 05:42:50 DEBUG ] 05:42:50 DEBUG } 05:42:50 DEBUG } 05:42:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760970" 05:42:50 INFO [http_cmd]: http status code OK 05:42:50 DEBUG --- status code --- 05:42:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:50 DEBUG --- http response --- 05:42:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:50 DEBUG ----- output ----- 05:42:50 DEBUG { 05:42:50 DEBUG "status": "success", 05:42:50 DEBUG "data": { 05:42:50 DEBUG "resultType": "vector", 05:42:50 DEBUG "result": [ 05:42:50 DEBUG { 05:42:50 DEBUG "metric": { 05:42:50 DEBUG "instance": "10.142.0.107:9100" 05:42:50 DEBUG }, 05:42:50 DEBUG "value": [ 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG [...] 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG "instance": "10.142.0.43:9100" 05:42:50 DEBUG }, 05:42:50 DEBUG "value": [ 05:42:50 DEBUG 1713760970, 05:42:50 DEBUG "0.0006666666666660604" 05:42:50 DEBUG ] 05:42:50 DEBUG } 05:42:50 DEBUG ] 05:42:50 DEBUG } 05:42:50 DEBUG } 05:42:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713760970" 05:42:50 INFO [http_cmd]: http status code OK 05:42:50 DEBUG --- status code --- 05:42:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:50 DEBUG --- http response --- 05:42:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:50 DEBUG ----- output ----- 05:42:50 DEBUG { 05:42:50 DEBUG "status": "success", 05:42:50 DEBUG "data": { 05:42:50 DEBUG "resultType": "vector", 05:42:50 DEBUG "result": [ 05:42:50 DEBUG { 05:42:50 DEBUG "metric": { 05:42:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:42:50 DEBUG }, 05:42:50 DEBUG "value": [ 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG [...] 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG 05:42:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:42:50 DEBUG }, 05:42:50 DEBUG "value": [ 05:42:50 DEBUG 1713760970, 05:42:50 DEBUG "7905341440" 05:42:50 DEBUG ] 05:42:50 DEBUG } 05:42:50 DEBUG ] 05:42:50 DEBUG } 05:42:50 DEBUG } 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760971" 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713760971" 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713760971" 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760971" 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG --- http response --- 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG --- http response --- 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG { 05:42:51 DEBUG { 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG "data": { 05:42:51 DEBUG "data": { 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG "result": [ 05:42:51 DEBUG "result": [ 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 DEBUG { 05:42:51 DEBUG { 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG "metric": { 05:42:51 DEBUG "metric": { 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:42:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:42:51 DEBUG --- http response --- 05:42:51 DEBUG }, 05:42:51 DEBUG }, 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:51 DEBUG "value": [ 05:42:51 DEBUG "value": [ 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG { 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "data": { 05:42:51 DEBUG [...] 05:42:51 DEBUG [...] 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "result": [ 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG { 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "metric": { 05:42:51 DEBUG "pod": "ds-cts-0" 05:42:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:42:51 DEBUG "pod": "ds-idrepo-0" 05:42:51 DEBUG }, 05:42:51 DEBUG }, 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG "value": [ 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG 05:42:51 DEBUG "35484352.511999995" 05:42:51 DEBUG "1.0496219064291739" 05:42:51 DEBUG 05:42:51 DEBUG ] 05:42:51 DEBUG ] 05:42:51 DEBUG 05:42:51 DEBUG } 05:42:51 DEBUG } 05:42:51 DEBUG [...] 05:42:51 DEBUG ] 05:42:51 DEBUG ] 05:42:51 DEBUG 05:42:51 DEBUG } 05:42:51 DEBUG } 05:42:51 DEBUG 05:42:51 DEBUG } 05:42:51 DEBUG } 05:42:51 DEBUG 05:42:51 DEBUG "pod": "ds-cts-2" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG "0" 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG } 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG --- http response --- 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG { 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG "data": { 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG "result": [ 05:42:51 DEBUG { 05:42:51 DEBUG "metric": { 05:42:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG [...] 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG "0" 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG } 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760971" 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG --- http response --- 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG { 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG "data": { 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG "result": [ 05:42:51 DEBUG { 05:42:51 DEBUG "metric": { 05:42:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG [...] 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG "0" 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG } 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760971" 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760971" 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG --- http response --- 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG { 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG "data": { 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG "result": [ 05:42:51 DEBUG { 05:42:51 DEBUG "metric": { 05:42:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG [...] 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG "0" 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG } 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760971" 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG --- http response --- 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG { 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG "data": { 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG "result": [ 05:42:51 DEBUG { 05:42:51 DEBUG "metric": { 05:42:51 DEBUG "pod": "ds-idrepo-0" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG [...] 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "pod": "ds-idrepo-2" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG "0" 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG } 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760971" 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713760971" 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713760971" 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG --- http response --- 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG { 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG "data": { 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG "result": [ 05:42:51 DEBUG { 05:42:51 DEBUG "metric": { 05:42:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG [...] 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG "0" 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG } 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG --- http response --- 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG { 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG --- http response --- 05:42:51 DEBUG "data": { 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG "result": [ 05:42:51 DEBUG { 05:42:51 DEBUG "metric": { 05:42:51 DEBUG "pod": "ds-cts-0" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG [...] 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "pod": "ds-cts-2" 05:42:51 DEBUG }, 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG "0" 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG } 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:51 INFO [http_cmd]: http status code OK 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG --- status code --- 05:42:51 DEBUG { 05:42:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG --- http response --- 05:42:51 DEBUG "data": { 05:42:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG ----- output ----- 05:42:51 DEBUG "result": [ 05:42:51 DEBUG { 05:42:51 DEBUG { 05:42:51 DEBUG "status": "success", 05:42:51 DEBUG "metric": { 05:42:51 DEBUG "data": { 05:42:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:42:51 DEBUG "resultType": "vector", 05:42:51 DEBUG }, 05:42:51 DEBUG "result": [ 05:42:51 DEBUG "value": [ 05:42:51 DEBUG { 05:42:51 DEBUG 05:42:51 DEBUG "metric": { 05:42:51 DEBUG 05:42:51 DEBUG "instance": "10.142.0.107:9100" 05:42:51 DEBUG 05:42:51 DEBUG }, 05:42:51 DEBUG [...] 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG 05:42:51 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:42:51 DEBUG [...] 05:42:51 DEBUG }, 05:42:51 DEBUG 05:42:51 DEBUG "value": [ 05:42:51 DEBUG 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG 05:42:51 DEBUG "0" 05:42:51 DEBUG "instance": "10.142.0.43:9100" 05:42:51 DEBUG ] 05:42:51 DEBUG }, 05:42:51 DEBUG } 05:42:51 DEBUG "value": [ 05:42:51 DEBUG ] 05:42:51 DEBUG 1713760971, 05:42:51 DEBUG } 05:42:51 DEBUG "1381843.5" 05:42:51 DEBUG } 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG ] 05:42:51 DEBUG } 05:42:51 DEBUG } 05:43:08 INFO 05:43:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:43:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:43:08 INFO [loop_until]: OK (rc = 0) 05:43:08 DEBUG --- stdout --- 05:43:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 645m 8790Mi am-767f6b67b5-qcqz2 686m 8839Mi am-767f6b67b5-vbltz 601m 8165Mi ds-cts-0 18182m 12511Mi ds-cts-1 187m 11385Mi ds-cts-2 18472m 12549Mi ds-idrepo-0 15m 4010Mi ds-idrepo-1 12m 4001Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1243Mi lodemon-7f68c498d9-gbc5j 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1005m 7747Mi 05:43:08 DEBUG --- stderr --- 05:43:08 DEBUG 05:43:16 INFO 05:43:16 INFO [loop_until]: kubectl --namespace=xlou top node 05:43:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:43:16 INFO [loop_until]: OK (rc = 0) 05:43:16 DEBUG --- stdout --- 05:43:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6243Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 8106Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 765m 2% 11516Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3353Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18241m 61% 15764Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 5221Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 767m 2% 11725Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 74m 0% 3134Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 682m 2% 10420Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 7476Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 206m 0% 16563Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 93m 0% 8047Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18652m 62% 15264Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1078m 6% 17885Mi 30% 05:43:16 DEBUG --- stderr --- 05:43:16 DEBUG 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761027" 05:43:47 ERROR [http_cmd]: ERROR 05:43:47 ERROR --- status code --- 05:43:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:43:47 ERROR --- http response --- 05:43:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:43:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761027" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761027" 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "data": { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 1713761027, 05:43:47 DEBUG "17501.93617305381" 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761027" 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "data": { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "pod": "ds-cts-0" 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "pod": "ds-cts-2" 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 1713761027, 05:43:47 DEBUG "75.9736" 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761027" 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "data": { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "pod": "ds-idrepo-0" 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "pod": "ds-idrepo-2" 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 1713761027, 05:43:47 DEBUG "0" 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761028" 05:43:48 INFO [http_cmd]: http status code OK 05:43:48 DEBUG --- status code --- 05:43:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:48 DEBUG --- http response --- 05:43:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:48 DEBUG ----- output ----- 05:43:48 DEBUG { 05:43:48 DEBUG "status": "success", 05:43:48 DEBUG "data": { 05:43:48 DEBUG "resultType": "vector", 05:43:48 DEBUG "result": [ 05:43:48 DEBUG { 05:43:48 DEBUG "metric": { 05:43:48 DEBUG "instance": "10.142.0.107:9100" 05:43:48 DEBUG }, 05:43:48 DEBUG "value": [ 05:43:48 DEBUG 05:43:48 DEBUG 05:43:48 DEBUG 05:43:48 DEBUG [...] 05:43:48 DEBUG 05:43:48 DEBUG 05:43:48 DEBUG 05:43:48 DEBUG "instance": "10.142.0.43:9100" 05:43:48 DEBUG }, 05:43:48 DEBUG "value": [ 05:43:48 DEBUG 1713761028, 05:43:48 DEBUG "5832691467264" 05:43:48 DEBUG ] 05:43:48 DEBUG } 05:43:48 DEBUG ] 05:43:48 DEBUG } 05:43:48 DEBUG } 05:43:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761029" 05:43:49 INFO [http_cmd]: http status code OK 05:43:49 DEBUG --- status code --- 05:43:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:49 DEBUG --- http response --- 05:43:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:49 DEBUG ----- output ----- 05:43:49 DEBUG { 05:43:49 DEBUG "status": "success", 05:43:49 DEBUG "data": { 05:43:49 DEBUG "resultType": "vector", 05:43:49 DEBUG "result": [ 05:43:49 DEBUG { 05:43:49 DEBUG "metric": { 05:43:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG [...] 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 1713761029, 05:43:49 DEBUG "500" 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG } 05:43:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761029" 05:43:49 INFO [http_cmd]: http status code OK 05:43:49 DEBUG --- status code --- 05:43:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:49 DEBUG --- http response --- 05:43:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:49 DEBUG ----- output ----- 05:43:49 DEBUG { 05:43:49 DEBUG "status": "success", 05:43:49 DEBUG "data": { 05:43:49 DEBUG "resultType": "vector", 05:43:49 DEBUG "result": [ 05:43:49 DEBUG { 05:43:49 DEBUG "metric": { 05:43:49 DEBUG "pod": "ds-cts-0" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG [...] 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG "pod": "ds-cts-2" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 1713761029, 05:43:49 DEBUG "28.3" 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG } 05:43:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761029" 05:43:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761029" 05:43:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761029" 05:43:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761029" 05:43:49 INFO [http_cmd]: http status code OK 05:43:49 DEBUG --- status code --- 05:43:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:49 DEBUG --- http response --- 05:43:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:49 DEBUG ----- output ----- 05:43:49 DEBUG { 05:43:49 DEBUG "status": "success", 05:43:49 DEBUG "data": { 05:43:49 DEBUG "resultType": "vector", 05:43:49 DEBUG "result": [ 05:43:49 DEBUG { 05:43:49 DEBUG "metric": { 05:43:49 DEBUG "pod": "ds-idrepo-0" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG [...] 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG "pod": "ds-idrepo-2" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 1713761029, 05:43:49 DEBUG "0" 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG } 05:43:49 INFO [http_cmd]: http status code OK 05:43:49 DEBUG --- status code --- 05:43:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:49 DEBUG --- http response --- 05:43:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:49 DEBUG ----- output ----- 05:43:49 DEBUG { 05:43:49 DEBUG "status": "success", 05:43:49 DEBUG "data": { 05:43:49 DEBUG "resultType": "vector", 05:43:49 DEBUG "result": [ 05:43:49 DEBUG { 05:43:49 DEBUG "metric": { 05:43:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG [...] 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 1713761029, 05:43:49 DEBUG "0" 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG } 05:43:49 INFO [http_cmd]: http status code OK 05:43:49 INFO [http_cmd]: http status code OK 05:43:49 DEBUG --- status code --- 05:43:49 DEBUG --- status code --- 05:43:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:49 DEBUG --- http response --- 05:43:49 DEBUG --- http response --- 05:43:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:49 DEBUG ----- output ----- 05:43:49 DEBUG { 05:43:49 DEBUG "status": "success", 05:43:49 DEBUG "data": { 05:43:49 DEBUG "resultType": "vector", 05:43:49 DEBUG "result": [ 05:43:49 DEBUG { 05:43:49 DEBUG "metric": { 05:43:49 DEBUG "instance": "10.142.0.107:9100" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG [...] 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG "instance": "10.142.0.43:9100" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 1713761029, 05:43:49 DEBUG "0.000599999999898652" 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG } 05:43:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:49 DEBUG ----- output ----- 05:43:49 DEBUG { 05:43:49 DEBUG "status": "success", 05:43:49 DEBUG "data": { 05:43:49 DEBUG "resultType": "vector", 05:43:49 DEBUG "result": [ 05:43:49 DEBUG { 05:43:49 DEBUG "metric": { 05:43:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG [...] 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 1713761029, 05:43:49 DEBUG "990.472" 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG } 05:43:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761029" 05:43:49 INFO [http_cmd]: http status code OK 05:43:49 DEBUG --- status code --- 05:43:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:49 DEBUG --- http response --- 05:43:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:43:49 DEBUG ----- output ----- 05:43:49 DEBUG { 05:43:49 DEBUG "status": "success", 05:43:49 DEBUG "data": { 05:43:49 DEBUG "resultType": "vector", 05:43:49 DEBUG "result": [ 05:43:49 DEBUG { 05:43:49 DEBUG "metric": { 05:43:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG [...] 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG 05:43:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:49 DEBUG }, 05:43:49 DEBUG "value": [ 05:43:49 DEBUG 1713761029, 05:43:49 DEBUG "546.9355054079316" 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG ] 05:43:49 DEBUG } 05:43:49 DEBUG } 05:43:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761030" 05:43:50 INFO [http_cmd]: http status code OK 05:43:50 DEBUG --- status code --- 05:43:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:50 DEBUG --- http response --- 05:43:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:50 DEBUG ----- output ----- 05:43:50 DEBUG { 05:43:50 DEBUG "status": "success", 05:43:50 DEBUG "data": { 05:43:50 DEBUG "resultType": "vector", 05:43:50 DEBUG "result": [ 05:43:50 DEBUG { 05:43:50 DEBUG "metric": { 05:43:50 DEBUG "instance": "10.142.0.107:9100" 05:43:50 DEBUG }, 05:43:50 DEBUG "value": [ 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG [...] 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG "instance": "10.142.0.43:9100" 05:43:50 DEBUG }, 05:43:50 DEBUG "value": [ 05:43:50 DEBUG 1713761030, 05:43:50 DEBUG "1272254.7" 05:43:50 DEBUG ] 05:43:50 DEBUG } 05:43:50 DEBUG ] 05:43:50 DEBUG } 05:43:50 DEBUG } 05:43:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761030" 05:43:50 INFO [http_cmd]: http status code OK 05:43:50 DEBUG --- status code --- 05:43:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:50 DEBUG --- http response --- 05:43:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:50 DEBUG ----- output ----- 05:43:50 DEBUG { 05:43:50 DEBUG "status": "success", 05:43:50 DEBUG "data": { 05:43:50 DEBUG "resultType": "vector", 05:43:50 DEBUG "result": [ 05:43:50 DEBUG { 05:43:50 DEBUG "metric": { 05:43:50 DEBUG "instance": "10.142.0.107:9100" 05:43:50 DEBUG }, 05:43:50 DEBUG "value": [ 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG [...] 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG "instance": "10.142.0.43:9100" 05:43:50 DEBUG }, 05:43:50 DEBUG "value": [ 05:43:50 DEBUG 1713761030, 05:43:50 DEBUG "0" 05:43:50 DEBUG ] 05:43:50 DEBUG } 05:43:50 DEBUG ] 05:43:50 DEBUG } 05:43:50 DEBUG } 05:43:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761030" 05:43:50 INFO [http_cmd]: http status code OK 05:43:50 DEBUG --- status code --- 05:43:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:50 DEBUG --- http response --- 05:43:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:50 DEBUG ----- output ----- 05:43:50 DEBUG { 05:43:50 DEBUG "status": "success", 05:43:50 DEBUG "data": { 05:43:50 DEBUG "resultType": "vector", 05:43:50 DEBUG "result": [ 05:43:50 DEBUG { 05:43:50 DEBUG "metric": { 05:43:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:43:50 DEBUG }, 05:43:50 DEBUG "value": [ 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG [...] 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG 05:43:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:43:50 DEBUG }, 05:43:50 DEBUG "value": [ 05:43:50 DEBUG 1713761030, 05:43:50 DEBUG "8832196608" 05:43:50 DEBUG ] 05:43:50 DEBUG } 05:43:50 DEBUG ] 05:43:50 DEBUG } 05:43:50 DEBUG } 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761031" 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 DEBUG "pod": "ds-idrepo-0" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "pod": "ds-cts-2" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "0" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761031" 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "0" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761031" 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761031" 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761031" 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "0" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "1.0318468058305161" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "0" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761031" 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761031" 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG }, 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG "value": [ 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "0" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 DEBUG "instance": "10.142.0.107:9100" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "instance": "10.142.0.43:9100" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "1266175.0999999999" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761031" 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761031" 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "0.6500670984141167" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761031" 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "0" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 DEBUG "pod": "ds-idrepo-0" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "pod": "ds-idrepo-2" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "0" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:43:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761031" 05:43:51 INFO [http_cmd]: http status code OK 05:43:51 DEBUG --- status code --- 05:43:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:51 DEBUG --- http response --- 05:43:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:51 DEBUG ----- output ----- 05:43:51 DEBUG { 05:43:51 DEBUG "status": "success", 05:43:51 DEBUG "data": { 05:43:51 DEBUG "resultType": "vector", 05:43:51 DEBUG "result": [ 05:43:51 DEBUG { 05:43:51 DEBUG "metric": { 05:43:51 DEBUG "pod": "ds-cts-0" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG [...] 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG 05:43:51 DEBUG "pod": "ds-cts-2" 05:43:51 DEBUG }, 05:43:51 DEBUG "value": [ 05:43:51 DEBUG 1713761031, 05:43:51 DEBUG "0" 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG ] 05:43:51 DEBUG } 05:43:51 DEBUG } 05:44:08 INFO 05:44:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:44:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:44:08 INFO [loop_until]: OK (rc = 0) 05:44:08 DEBUG --- stdout --- 05:44:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 754m 8791Mi am-767f6b67b5-qcqz2 810m 8841Mi am-767f6b67b5-vbltz 756m 8167Mi ds-cts-0 17364m 11038Mi ds-cts-1 188m 11385Mi ds-cts-2 19090m 12548Mi ds-idrepo-0 31m 4009Mi ds-idrepo-1 13m 4001Mi ds-idrepo-2 12m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1243Mi lodemon-7f68c498d9-gbc5j 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1014m 8504Mi 05:44:08 DEBUG --- stderr --- 05:44:08 DEBUG 05:44:16 INFO 05:44:16 INFO [loop_until]: kubectl --namespace=xlou top node 05:44:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:44:16 INFO [loop_until]: OK (rc = 0) 05:44:16 DEBUG --- stdout --- 05:44:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6223Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 94m 0% 8108Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 809m 2% 11519Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3359Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17027m 57% 12500Mi 11% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 912m 3% 11722Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 72m 0% 3132Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 804m 2% 10417Mi 9% gke-xlou-cdm-ds-0f601751-j488 69m 0% 7490Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 242m 0% 16551Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 90m 0% 8043Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18734m 62% 15270Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1084m 6% 18802Mi 32% 05:44:16 DEBUG --- stderr --- 05:44:16 DEBUG 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761087" 05:44:47 ERROR [http_cmd]: ERROR 05:44:47 ERROR --- status code --- 05:44:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:44:47 ERROR --- http response --- 05:44:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:44:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761087" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761087" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761087" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761087" 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG --- status code --- 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "data": { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "result": [ 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "pod": "ds-cts-0" 05:44:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:44:47 DEBUG }, 05:44:47 DEBUG }, 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "pod": "ds-cts-2" 05:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:47 DEBUG }, 05:44:47 DEBUG }, 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 1713761087, 05:44:47 DEBUG 1713761087, 05:44:47 DEBUG "73.6581" 05:44:47 DEBUG "280322919.10914606" 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG { 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "data": { 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "result": [ 05:44:47 DEBUG { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "pod": "ds-idrepo-0" 05:44:47 DEBUG }, 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "pod": "ds-idrepo-2" 05:44:47 DEBUG }, 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 1713761087, 05:44:47 DEBUG "0" 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761088" 05:44:48 INFO [http_cmd]: http status code OK 05:44:48 DEBUG --- status code --- 05:44:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:48 DEBUG --- http response --- 05:44:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:48 DEBUG ----- output ----- 05:44:48 DEBUG { 05:44:48 DEBUG "status": "success", 05:44:48 DEBUG "data": { 05:44:48 DEBUG "resultType": "vector", 05:44:48 DEBUG "result": [ 05:44:48 DEBUG { 05:44:48 DEBUG "metric": { 05:44:48 DEBUG "instance": "10.142.0.107:9100" 05:44:48 DEBUG }, 05:44:48 DEBUG "value": [ 05:44:48 DEBUG 05:44:48 DEBUG 05:44:48 DEBUG 05:44:48 DEBUG [...] 05:44:48 DEBUG 05:44:48 DEBUG 05:44:48 DEBUG 05:44:48 DEBUG "instance": "10.142.0.43:9100" 05:44:48 DEBUG }, 05:44:48 DEBUG "value": [ 05:44:48 DEBUG 1713761088, 05:44:48 DEBUG "5832692163584" 05:44:48 DEBUG ] 05:44:48 DEBUG } 05:44:48 DEBUG ] 05:44:48 DEBUG } 05:44:48 DEBUG } 05:44:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761089" 05:44:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761089" 05:44:49 INFO [http_cmd]: http status code OK 05:44:49 DEBUG --- status code --- 05:44:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:49 DEBUG --- http response --- 05:44:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:49 DEBUG ----- output ----- 05:44:49 DEBUG { 05:44:49 DEBUG "status": "success", 05:44:49 DEBUG "data": { 05:44:49 DEBUG "resultType": "vector", 05:44:49 DEBUG "result": [ 05:44:49 DEBUG { 05:44:49 DEBUG "metric": { 05:44:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG [...] 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 1713761089, 05:44:49 DEBUG "333.3333333333333" 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG } 05:44:49 INFO [http_cmd]: http status code OK 05:44:49 DEBUG --- status code --- 05:44:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:49 DEBUG --- http response --- 05:44:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:49 DEBUG ----- output ----- 05:44:49 DEBUG { 05:44:49 DEBUG "status": "success", 05:44:49 DEBUG "data": { 05:44:49 DEBUG "resultType": "vector", 05:44:49 DEBUG "result": [ 05:44:49 DEBUG { 05:44:49 DEBUG "metric": { 05:44:49 DEBUG "pod": "ds-cts-0" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG [...] 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG "pod": "ds-cts-2" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 1713761089, 05:44:49 DEBUG "28.933333333333334" 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG } 05:44:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761089" 05:44:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761089" 05:44:49 INFO [http_cmd]: http status code OK 05:44:49 DEBUG --- status code --- 05:44:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:49 DEBUG --- http response --- 05:44:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:49 DEBUG ----- output ----- 05:44:49 DEBUG { 05:44:49 DEBUG "status": "success", 05:44:49 DEBUG "data": { 05:44:49 DEBUG "resultType": "vector", 05:44:49 DEBUG "result": [ 05:44:49 DEBUG { 05:44:49 DEBUG "metric": { 05:44:49 DEBUG "instance": "10.142.0.107:9100" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG [...] 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG "instance": "10.142.0.43:9100" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 1713761089, 05:44:49 DEBUG "0.0006333333333600422" 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG } 05:44:49 INFO [http_cmd]: http status code OK 05:44:49 DEBUG --- status code --- 05:44:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:49 DEBUG --- http response --- 05:44:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:49 DEBUG ----- output ----- 05:44:49 DEBUG { 05:44:49 DEBUG "status": "success", 05:44:49 DEBUG "data": { 05:44:49 DEBUG "resultType": "vector", 05:44:49 DEBUG "result": [ 05:44:49 DEBUG { 05:44:49 DEBUG "metric": { 05:44:49 DEBUG "pod": "ds-idrepo-0" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG [...] 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG "pod": "ds-idrepo-2" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 1713761089, 05:44:49 DEBUG "0" 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG } 05:44:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761089" 05:44:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761089" 05:44:49 INFO [http_cmd]: http status code OK 05:44:49 DEBUG --- status code --- 05:44:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:49 DEBUG --- http response --- 05:44:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:49 DEBUG ----- output ----- 05:44:49 DEBUG { 05:44:49 DEBUG "status": "success", 05:44:49 DEBUG "data": { 05:44:49 DEBUG "resultType": "vector", 05:44:49 DEBUG "result": [ 05:44:49 DEBUG { 05:44:49 DEBUG "metric": { 05:44:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG [...] 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 1713761089, 05:44:49 DEBUG "0" 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG } 05:44:49 INFO [http_cmd]: http status code OK 05:44:49 DEBUG --- status code --- 05:44:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:49 DEBUG --- http response --- 05:44:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:49 DEBUG ----- output ----- 05:44:49 DEBUG { 05:44:49 DEBUG "status": "success", 05:44:49 DEBUG "data": { 05:44:49 DEBUG "resultType": "vector", 05:44:49 DEBUG "result": [ 05:44:49 DEBUG { 05:44:49 DEBUG "metric": { 05:44:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG [...] 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 1713761089, 05:44:49 DEBUG "991.21" 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG } 05:44:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761089" 05:44:49 INFO [http_cmd]: http status code OK 05:44:49 DEBUG --- status code --- 05:44:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:49 DEBUG --- http response --- 05:44:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:44:49 DEBUG ----- output ----- 05:44:49 DEBUG { 05:44:49 DEBUG "status": "success", 05:44:49 DEBUG "data": { 05:44:49 DEBUG "resultType": "vector", 05:44:49 DEBUG "result": [ 05:44:49 DEBUG { 05:44:49 DEBUG "metric": { 05:44:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG [...] 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG 05:44:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:49 DEBUG }, 05:44:49 DEBUG "value": [ 05:44:49 DEBUG 1713761089, 05:44:49 DEBUG "508.86329364082474" 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG ] 05:44:49 DEBUG } 05:44:49 DEBUG } 05:44:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761090" 05:44:50 INFO [http_cmd]: http status code OK 05:44:50 DEBUG --- status code --- 05:44:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:50 DEBUG --- http response --- 05:44:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:50 DEBUG ----- output ----- 05:44:50 DEBUG { 05:44:50 DEBUG "status": "success", 05:44:50 DEBUG "data": { 05:44:50 DEBUG "resultType": "vector", 05:44:50 DEBUG "result": [ 05:44:50 DEBUG { 05:44:50 DEBUG "metric": { 05:44:50 DEBUG "instance": "10.142.0.107:9100" 05:44:50 DEBUG }, 05:44:50 DEBUG "value": [ 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG [...] 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG "instance": "10.142.0.43:9100" 05:44:50 DEBUG }, 05:44:50 DEBUG "value": [ 05:44:50 DEBUG 1713761090, 05:44:50 DEBUG "484315.5" 05:44:50 DEBUG ] 05:44:50 DEBUG } 05:44:50 DEBUG ] 05:44:50 DEBUG } 05:44:50 DEBUG } 05:44:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761090" 05:44:50 INFO [http_cmd]: http status code OK 05:44:50 DEBUG --- status code --- 05:44:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:50 DEBUG --- http response --- 05:44:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:50 DEBUG ----- output ----- 05:44:50 DEBUG { 05:44:50 DEBUG "status": "success", 05:44:50 DEBUG "data": { 05:44:50 DEBUG "resultType": "vector", 05:44:50 DEBUG "result": [ 05:44:50 DEBUG { 05:44:50 DEBUG "metric": { 05:44:50 DEBUG "instance": "10.142.0.107:9100" 05:44:50 DEBUG }, 05:44:50 DEBUG "value": [ 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG [...] 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG "instance": "10.142.0.43:9100" 05:44:50 DEBUG }, 05:44:50 DEBUG "value": [ 05:44:50 DEBUG 1713761090, 05:44:50 DEBUG "0" 05:44:50 DEBUG ] 05:44:50 DEBUG } 05:44:50 DEBUG ] 05:44:50 DEBUG } 05:44:50 DEBUG } 05:44:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761090" 05:44:50 INFO [http_cmd]: http status code OK 05:44:50 DEBUG --- status code --- 05:44:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:50 DEBUG --- http response --- 05:44:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:50 DEBUG ----- output ----- 05:44:50 DEBUG { 05:44:50 DEBUG "status": "success", 05:44:50 DEBUG "data": { 05:44:50 DEBUG "resultType": "vector", 05:44:50 DEBUG "result": [ 05:44:50 DEBUG { 05:44:50 DEBUG "metric": { 05:44:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:44:50 DEBUG }, 05:44:50 DEBUG "value": [ 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG [...] 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG 05:44:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:44:50 DEBUG }, 05:44:50 DEBUG "value": [ 05:44:50 DEBUG 1713761090, 05:44:50 DEBUG "9746694144" 05:44:50 DEBUG ] 05:44:50 DEBUG } 05:44:50 DEBUG ] 05:44:50 DEBUG } 05:44:50 DEBUG } 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761091" 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 DEBUG --- status code --- 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG { 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG "data": { 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG "result": [ 05:44:51 DEBUG { 05:44:51 DEBUG "metric": { 05:44:51 DEBUG "pod": "ds-idrepo-0" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG [...] 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG "pod": "ds-cts-2" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG "0" 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761091" 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761091" 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761091" 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 DEBUG --- status code --- 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 DEBUG --- status code --- 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG --- status code --- 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG { 05:44:51 DEBUG { 05:44:51 DEBUG { 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG "data": { 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG "data": { 05:44:51 DEBUG "data": { 05:44:51 DEBUG "result": [ 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG { 05:44:51 DEBUG "result": [ 05:44:51 DEBUG "result": [ 05:44:51 DEBUG "metric": { 05:44:51 DEBUG { 05:44:51 DEBUG { 05:44:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:51 DEBUG "metric": { 05:44:51 DEBUG "metric": { 05:44:51 DEBUG }, 05:44:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:44:51 DEBUG "value": [ 05:44:51 DEBUG }, 05:44:51 DEBUG }, 05:44:51 DEBUG 05:44:51 DEBUG "value": [ 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG [...] 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG [...] 05:44:51 DEBUG [...] 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG }, 05:44:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:51 DEBUG "value": [ 05:44:51 DEBUG }, 05:44:51 DEBUG }, 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG "value": [ 05:44:51 DEBUG "0" 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG ] 05:44:51 DEBUG "0" 05:44:51 DEBUG "0" 05:44:51 DEBUG } 05:44:51 DEBUG ] 05:44:51 DEBUG ] 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 DEBUG ] 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761091" 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761091" 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761091" 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 DEBUG --- status code --- 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG --- status code --- 05:44:51 DEBUG { 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG "data": { 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG "result": [ 05:44:51 DEBUG { 05:44:51 DEBUG "metric": { 05:44:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG [...] 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG "0" 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG { 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG "data": { 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG "result": [ 05:44:51 DEBUG { 05:44:51 DEBUG "metric": { 05:44:51 DEBUG "instance": "10.142.0.107:9100" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG [...] 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG "instance": "10.142.0.43:9100" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG "478993.4333333333" 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 DEBUG --- status code --- 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG { 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG "data": { 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG "result": [ 05:44:51 DEBUG { 05:44:51 DEBUG "metric": { 05:44:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG [...] 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG "1.028926199838081" 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761091" 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 DEBUG --- status code --- 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG { 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG "data": { 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG "result": [ 05:44:51 DEBUG { 05:44:51 DEBUG "metric": { 05:44:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG [...] 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG "0.3111355682064909" 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761091" 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761091" 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 DEBUG --- status code --- 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG { 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG "data": { 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG "result": [ 05:44:51 DEBUG { 05:44:51 DEBUG "metric": { 05:44:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG [...] 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG "0" 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 DEBUG --- status code --- 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG { 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG "data": { 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG "result": [ 05:44:51 DEBUG { 05:44:51 DEBUG "metric": { 05:44:51 DEBUG "pod": "ds-idrepo-0" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG [...] 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG "pod": "ds-idrepo-2" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG "0" 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG } 05:44:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761091" 05:44:51 INFO [http_cmd]: http status code OK 05:44:51 DEBUG --- status code --- 05:44:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:51 DEBUG --- http response --- 05:44:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:51 DEBUG ----- output ----- 05:44:51 DEBUG { 05:44:51 DEBUG "status": "success", 05:44:51 DEBUG "data": { 05:44:51 DEBUG "resultType": "vector", 05:44:51 DEBUG "result": [ 05:44:51 DEBUG { 05:44:51 DEBUG "metric": { 05:44:51 DEBUG "pod": "ds-cts-0" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG [...] 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG 05:44:51 DEBUG "pod": "ds-cts-2" 05:44:51 DEBUG }, 05:44:51 DEBUG "value": [ 05:44:51 DEBUG 1713761091, 05:44:51 DEBUG "0" 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG ] 05:44:51 DEBUG } 05:44:51 DEBUG } 05:45:08 INFO 05:45:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:45:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:45:08 INFO [loop_until]: OK (rc = 0) 05:45:08 DEBUG --- stdout --- 05:45:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 383m 8791Mi am-767f6b67b5-qcqz2 428m 8840Mi am-767f6b67b5-vbltz 333m 8167Mi ds-cts-0 100m 5290Mi ds-cts-1 114m 11384Mi ds-cts-2 19429m 12551Mi ds-idrepo-0 14m 4009Mi ds-idrepo-1 15m 3994Mi ds-idrepo-2 14m 3985Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1243Mi lodemon-7f68c498d9-gbc5j 1m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1054m 9493Mi 05:45:08 DEBUG --- stderr --- 05:45:08 DEBUG 05:45:16 INFO 05:45:16 INFO [loop_until]: kubectl --namespace=xlou top node 05:45:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:45:16 INFO [loop_until]: OK (rc = 0) 05:45:16 DEBUG --- stdout --- 05:45:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6221Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 8094Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 414m 1% 11528Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3366Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 175m 0% 8532Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5220Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 482m 1% 11728Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 3121Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 456m 1% 10418Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 7482Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 187m 0% 16547Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 8044Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19046m 63% 15285Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1042m 6% 19686Mi 33% 05:45:16 DEBUG --- stderr --- 05:45:16 DEBUG 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761147" 05:45:47 ERROR [http_cmd]: ERROR 05:45:47 ERROR --- status code --- 05:45:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:45:47 ERROR --- http response --- 05:45:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:45:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761147" 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761147" 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "data": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "result": [ 05:45:47 DEBUG { 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "pod": "ds-idrepo-0" 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "pod": "ds-idrepo-2" 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 1713761147, 05:45:47 DEBUG "0" 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761147" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761147" 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "data": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "result": [ 05:45:47 DEBUG { 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 1713761147, 05:45:47 DEBUG "214790809.7476029" 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "data": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "result": [ 05:45:47 DEBUG { 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "pod": "ds-cts-0" 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "pod": "ds-cts-2" 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 1713761147, 05:45:47 DEBUG "71.20066666666666" 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761148" 05:45:48 INFO [http_cmd]: http status code OK 05:45:48 DEBUG --- status code --- 05:45:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:48 DEBUG --- http response --- 05:45:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:48 DEBUG ----- output ----- 05:45:48 DEBUG { 05:45:48 DEBUG "status": "success", 05:45:48 DEBUG "data": { 05:45:48 DEBUG "resultType": "vector", 05:45:48 DEBUG "result": [ 05:45:48 DEBUG { 05:45:48 DEBUG "metric": { 05:45:48 DEBUG "instance": "10.142.0.107:9100" 05:45:48 DEBUG }, 05:45:48 DEBUG "value": [ 05:45:48 DEBUG 05:45:48 DEBUG 05:45:48 DEBUG 05:45:48 DEBUG [...] 05:45:48 DEBUG 05:45:48 DEBUG 05:45:48 DEBUG 05:45:48 DEBUG "instance": "10.142.0.43:9100" 05:45:48 DEBUG }, 05:45:48 DEBUG "value": [ 05:45:48 DEBUG 1713761148, 05:45:48 DEBUG "5832692876288" 05:45:48 DEBUG ] 05:45:48 DEBUG } 05:45:48 DEBUG ] 05:45:48 DEBUG } 05:45:48 DEBUG } 05:45:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761149" 05:45:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761149" 05:45:49 INFO [http_cmd]: http status code OK 05:45:49 DEBUG --- status code --- 05:45:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:49 DEBUG --- http response --- 05:45:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:49 DEBUG ----- output ----- 05:45:49 DEBUG { 05:45:49 DEBUG "status": "success", 05:45:49 DEBUG "data": { 05:45:49 DEBUG "resultType": "vector", 05:45:49 DEBUG "result": [ 05:45:49 DEBUG { 05:45:49 DEBUG "metric": { 05:45:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG [...] 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 1713761149, 05:45:49 DEBUG "166.66666666666666" 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG } 05:45:49 INFO [http_cmd]: http status code OK 05:45:49 DEBUG --- status code --- 05:45:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:49 DEBUG --- http response --- 05:45:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:49 DEBUG ----- output ----- 05:45:49 DEBUG { 05:45:49 DEBUG "status": "success", 05:45:49 DEBUG "data": { 05:45:49 DEBUG "resultType": "vector", 05:45:49 DEBUG "result": [ 05:45:49 DEBUG { 05:45:49 DEBUG "metric": { 05:45:49 DEBUG "pod": "ds-cts-0" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG [...] 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG "pod": "ds-cts-2" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 1713761149, 05:45:49 DEBUG "25.666666666666668" 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG } 05:45:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761149" 05:45:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761149" 05:45:49 INFO [http_cmd]: http status code OK 05:45:49 DEBUG --- status code --- 05:45:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:49 DEBUG --- http response --- 05:45:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:49 DEBUG ----- output ----- 05:45:49 DEBUG { 05:45:49 DEBUG "status": "success", 05:45:49 DEBUG "data": { 05:45:49 DEBUG "resultType": "vector", 05:45:49 DEBUG "result": [ 05:45:49 DEBUG { 05:45:49 DEBUG "metric": { 05:45:49 DEBUG "instance": "10.142.0.107:9100" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG [...] 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG "instance": "10.142.0.43:9100" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 1713761149, 05:45:49 DEBUG "0.0005666666668086389" 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG } 05:45:49 INFO [http_cmd]: http status code OK 05:45:49 DEBUG --- status code --- 05:45:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:49 DEBUG --- http response --- 05:45:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:49 DEBUG ----- output ----- 05:45:49 DEBUG { 05:45:49 DEBUG "status": "success", 05:45:49 DEBUG "data": { 05:45:49 DEBUG "resultType": "vector", 05:45:49 DEBUG "result": [ 05:45:49 DEBUG { 05:45:49 DEBUG "metric": { 05:45:49 DEBUG "pod": "ds-idrepo-0" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG [...] 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG "pod": "ds-idrepo-2" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 1713761149, 05:45:49 DEBUG "0" 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG } 05:45:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761149" 05:45:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761149" 05:45:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761149" 05:45:49 INFO [http_cmd]: http status code OK 05:45:49 DEBUG --- status code --- 05:45:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:49 DEBUG --- http response --- 05:45:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:49 DEBUG ----- output ----- 05:45:49 DEBUG { 05:45:49 DEBUG "status": "success", 05:45:49 DEBUG "data": { 05:45:49 DEBUG "resultType": "vector", 05:45:49 DEBUG "result": [ 05:45:49 DEBUG { 05:45:49 DEBUG "metric": { 05:45:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG [...] 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 1713761149, 05:45:49 DEBUG "0" 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG } 05:45:49 INFO [http_cmd]: http status code OK 05:45:49 DEBUG --- status code --- 05:45:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:49 DEBUG --- http response --- 05:45:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:49 DEBUG ----- output ----- 05:45:49 DEBUG { 05:45:49 DEBUG "status": "success", 05:45:49 DEBUG "data": { 05:45:49 DEBUG "resultType": "vector", 05:45:49 DEBUG "result": [ 05:45:49 DEBUG { 05:45:49 DEBUG "metric": { 05:45:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG [...] 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 1713761149, 05:45:49 DEBUG "991.5155" 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG } 05:45:49 INFO [http_cmd]: http status code OK 05:45:49 DEBUG --- status code --- 05:45:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:49 DEBUG --- http response --- 05:45:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:49 DEBUG ----- output ----- 05:45:49 DEBUG { 05:45:49 DEBUG "status": "success", 05:45:49 DEBUG "data": { 05:45:49 DEBUG "resultType": "vector", 05:45:49 DEBUG "result": [ 05:45:49 DEBUG { 05:45:49 DEBUG "metric": { 05:45:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG [...] 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG 05:45:49 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:45:49 DEBUG }, 05:45:49 DEBUG "value": [ 05:45:49 DEBUG 1713761149, 05:45:49 DEBUG "0" 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG ] 05:45:49 DEBUG } 05:45:49 DEBUG } 05:45:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761150" 05:45:50 INFO [http_cmd]: http status code OK 05:45:50 DEBUG --- status code --- 05:45:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:50 DEBUG --- http response --- 05:45:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:50 DEBUG ----- output ----- 05:45:50 DEBUG { 05:45:50 DEBUG "status": "success", 05:45:50 DEBUG "data": { 05:45:50 DEBUG "resultType": "vector", 05:45:50 DEBUG "result": [ 05:45:50 DEBUG { 05:45:50 DEBUG "metric": { 05:45:50 DEBUG "instance": "10.142.0.107:9100" 05:45:50 DEBUG }, 05:45:50 DEBUG "value": [ 05:45:50 DEBUG 05:45:50 DEBUG 05:45:50 DEBUG 05:45:50 DEBUG [...] 05:45:50 DEBUG 05:45:50 DEBUG 05:45:50 DEBUG 05:45:50 DEBUG "instance": "10.142.0.43:9100" 05:45:50 DEBUG }, 05:45:50 DEBUG "value": [ 05:45:50 DEBUG 1713761150, 05:45:50 DEBUG "1265407.8333333333" 05:45:50 DEBUG ] 05:45:50 DEBUG } 05:45:50 DEBUG ] 05:45:50 DEBUG } 05:45:50 DEBUG } 05:45:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761150" 05:45:50 INFO [http_cmd]: http status code OK 05:45:50 DEBUG --- status code --- 05:45:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:50 DEBUG --- http response --- 05:45:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:50 DEBUG ----- output ----- 05:45:50 DEBUG { 05:45:50 DEBUG "status": "success", 05:45:50 DEBUG "data": { 05:45:50 DEBUG "resultType": "vector", 05:45:50 DEBUG "result": [ 05:45:50 DEBUG { 05:45:50 DEBUG "metric": { 05:45:50 DEBUG "instance": "10.142.0.107:9100" 05:45:50 DEBUG }, 05:45:50 DEBUG "value": [ 05:45:50 DEBUG 05:45:50 DEBUG 05:45:50 DEBUG 05:45:50 DEBUG [...] 05:45:50 DEBUG 05:45:50 DEBUG 05:45:50 DEBUG 05:45:50 DEBUG "instance": "10.142.0.43:9100" 05:45:50 DEBUG }, 05:45:50 DEBUG "value": [ 05:45:50 DEBUG 1713761150, 05:45:50 DEBUG "0" 05:45:50 DEBUG ] 05:45:50 DEBUG } 05:45:50 DEBUG ] 05:45:50 DEBUG } 05:45:50 DEBUG } 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761151" 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG { 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG [...] 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "10278301696" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761151" 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761151" 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG { 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "pod": "ds-idrepo-0" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG [...] 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "pod": "ds-cts-2" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "0" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG { 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG [...] 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "0" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761151" 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761151" 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761151" 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761151" 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG { 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG [...] 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "0" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG { 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG [...] 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "0" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG --- http response --- 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG { 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:51 DEBUG 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG 05:45:51 DEBUG { 05:45:51 DEBUG [...] 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG 05:45:51 DEBUG "data": { 05:45:51 DEBUG 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG 05:45:51 DEBUG "result": [ 05:45:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:45:51 DEBUG { 05:45:51 DEBUG }, 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "value": [ 05:45:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG }, 05:45:51 DEBUG "1.0269627506468304" 05:45:51 DEBUG "value": [ 05:45:51 DEBUG ] 05:45:51 DEBUG 05:45:51 DEBUG } 05:45:51 DEBUG 05:45:51 DEBUG ] 05:45:51 DEBUG 05:45:51 DEBUG } 05:45:51 DEBUG [...] 05:45:51 DEBUG } 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "0" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761151" 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761151" 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761151" 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG { 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "instance": "10.142.0.107:9100" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG [...] 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "instance": "10.142.0.43:9100" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "1258719.5" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG { 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "pod": "ds-cts-0" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG [...] 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "pod": "ds-cts-2" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "0" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG { 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG [...] 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "0.000022448827132811508" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761151" 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG { 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG [...] 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "0" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:45:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761151" 05:45:51 INFO [http_cmd]: http status code OK 05:45:51 DEBUG --- status code --- 05:45:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:51 DEBUG --- http response --- 05:45:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:51 DEBUG ----- output ----- 05:45:51 DEBUG { 05:45:51 DEBUG "status": "success", 05:45:51 DEBUG "data": { 05:45:51 DEBUG "resultType": "vector", 05:45:51 DEBUG "result": [ 05:45:51 DEBUG { 05:45:51 DEBUG "metric": { 05:45:51 DEBUG "pod": "ds-idrepo-0" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG [...] 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG 05:45:51 DEBUG "pod": "ds-idrepo-2" 05:45:51 DEBUG }, 05:45:51 DEBUG "value": [ 05:45:51 DEBUG 1713761151, 05:45:51 DEBUG "0" 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG ] 05:45:51 DEBUG } 05:45:51 DEBUG } 05:46:08 INFO 05:46:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:46:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:46:08 INFO [loop_until]: OK (rc = 0) 05:46:08 DEBUG --- stdout --- 05:46:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 341m 8792Mi am-767f6b67b5-qcqz2 374m 8840Mi am-767f6b67b5-vbltz 445m 8166Mi ds-cts-0 129m 5290Mi ds-cts-1 98m 11385Mi ds-cts-2 18584m 12545Mi ds-idrepo-0 14m 4010Mi ds-idrepo-1 14m 3992Mi ds-idrepo-2 12m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1244Mi lodemon-7f68c498d9-gbc5j 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1004m 9798Mi 05:46:08 DEBUG --- stderr --- 05:46:08 DEBUG 05:46:16 INFO 05:46:16 INFO [loop_until]: kubectl --namespace=xlou top node 05:46:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:46:16 INFO [loop_until]: OK (rc = 0) 05:46:16 DEBUG --- stdout --- 05:46:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6217Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 8098Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 342m 1% 11522Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3365Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 156m 0% 8534Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 459m 1% 11723Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 3124Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 513m 1% 10411Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 7475Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 138m 0% 16569Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 91m 0% 8038Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18770m 62% 15281Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 19942Mi 34% 05:46:16 DEBUG --- stderr --- 05:46:16 DEBUG 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761207" 05:46:47 ERROR [http_cmd]: ERROR 05:46:47 ERROR --- status code --- 05:46:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:46:47 ERROR --- http response --- 05:46:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:46:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761207" 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761207" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761207" 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG { 05:46:47 DEBUG { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "data": { 05:46:47 DEBUG "data": { 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "result": [ 05:46:47 DEBUG "result": [ 05:46:47 DEBUG { 05:46:47 DEBUG { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "pod": "ds-cts-0" 05:46:47 DEBUG "pod": "ds-idrepo-0" 05:46:47 DEBUG }, 05:46:47 DEBUG }, 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "pod": "ds-cts-2" 05:46:47 DEBUG "pod": "ds-idrepo-2" 05:46:47 DEBUG }, 05:46:47 DEBUG }, 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 1713761207, 05:46:47 DEBUG 1713761207, 05:46:47 DEBUG "63.76853333333334" 05:46:47 DEBUG "0" 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761207" 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "data": { 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "result": [ 05:46:47 DEBUG { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:46:47 DEBUG }, 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:46:47 DEBUG }, 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 1713761207, 05:46:47 DEBUG "298174167.10953647" 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761209" 05:46:49 INFO [http_cmd]: http status code OK 05:46:49 DEBUG --- status code --- 05:46:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:49 DEBUG --- http response --- 05:46:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:49 DEBUG ----- output ----- 05:46:49 DEBUG { 05:46:49 DEBUG "status": "success", 05:46:49 DEBUG "data": { 05:46:49 DEBUG "resultType": "vector", 05:46:49 DEBUG "result": [ 05:46:49 DEBUG { 05:46:49 DEBUG "metric": { 05:46:49 DEBUG "instance": "10.142.0.107:9100" 05:46:49 DEBUG }, 05:46:49 DEBUG "value": [ 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG [...] 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG "instance": "10.142.0.43:9100" 05:46:49 DEBUG }, 05:46:49 DEBUG "value": [ 05:46:49 DEBUG 1713761209, 05:46:49 DEBUG "5832693697536" 05:46:49 DEBUG ] 05:46:49 DEBUG } 05:46:49 DEBUG ] 05:46:49 DEBUG } 05:46:49 DEBUG } 05:46:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761209" 05:46:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761209" 05:46:49 INFO [http_cmd]: http status code OK 05:46:49 DEBUG --- status code --- 05:46:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:49 DEBUG --- http response --- 05:46:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:49 DEBUG ----- output ----- 05:46:49 DEBUG { 05:46:49 DEBUG "status": "success", 05:46:49 DEBUG "data": { 05:46:49 DEBUG "resultType": "vector", 05:46:49 DEBUG "result": [ 05:46:49 DEBUG { 05:46:49 DEBUG "metric": { 05:46:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:49 DEBUG }, 05:46:49 DEBUG "value": [ 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG [...] 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:49 DEBUG }, 05:46:49 DEBUG "value": [ 05:46:49 DEBUG 1713761209, 05:46:49 DEBUG "166.66666666666666" 05:46:49 DEBUG ] 05:46:49 DEBUG } 05:46:49 DEBUG ] 05:46:49 DEBUG } 05:46:49 DEBUG } 05:46:49 INFO [http_cmd]: http status code OK 05:46:49 DEBUG --- status code --- 05:46:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:49 DEBUG --- http response --- 05:46:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:49 DEBUG ----- output ----- 05:46:49 DEBUG { 05:46:49 DEBUG "status": "success", 05:46:49 DEBUG "data": { 05:46:49 DEBUG "resultType": "vector", 05:46:49 DEBUG "result": [ 05:46:49 DEBUG { 05:46:49 DEBUG "metric": { 05:46:49 DEBUG "pod": "ds-cts-0" 05:46:49 DEBUG }, 05:46:49 DEBUG "value": [ 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG [...] 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG 05:46:49 DEBUG "pod": "ds-cts-2" 05:46:49 DEBUG }, 05:46:49 DEBUG "value": [ 05:46:49 DEBUG 1713761209, 05:46:49 DEBUG "65.1" 05:46:49 DEBUG ] 05:46:49 DEBUG } 05:46:49 DEBUG ] 05:46:49 DEBUG } 05:46:49 DEBUG } 05:46:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761210" 05:46:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761210" 05:46:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761210" 05:46:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761210" 05:46:50 INFO [http_cmd]: http status code OK 05:46:50 INFO [http_cmd]: http status code OK 05:46:50 INFO [http_cmd]: http status code OK 05:46:50 DEBUG --- status code --- 05:46:50 DEBUG --- status code --- 05:46:50 DEBUG --- status code --- 05:46:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:50 DEBUG --- http response --- 05:46:50 DEBUG --- http response --- 05:46:50 DEBUG --- http response --- 05:46:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:50 DEBUG ----- output ----- 05:46:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:50 DEBUG ----- output ----- 05:46:50 DEBUG { 05:46:50 DEBUG ----- output ----- 05:46:50 DEBUG { 05:46:50 DEBUG "status": "success", 05:46:50 DEBUG { 05:46:50 DEBUG "status": "success", 05:46:50 DEBUG "data": { 05:46:50 DEBUG "status": "success", 05:46:50 DEBUG "data": { 05:46:50 DEBUG "resultType": "vector", 05:46:50 DEBUG "data": { 05:46:50 DEBUG "resultType": "vector", 05:46:50 DEBUG "result": [ 05:46:50 DEBUG "resultType": "vector", 05:46:50 DEBUG "result": [ 05:46:50 DEBUG { 05:46:50 DEBUG "result": [ 05:46:50 DEBUG { 05:46:50 DEBUG "metric": { 05:46:50 DEBUG { 05:46:50 DEBUG "metric": { 05:46:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:50 DEBUG "metric": { 05:46:50 DEBUG "pod": "ds-idrepo-0" 05:46:50 DEBUG }, 05:46:50 DEBUG "instance": "10.142.0.107:9100" 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG [...] 05:46:50 DEBUG 05:46:50 DEBUG [...] 05:46:50 DEBUG 05:46:50 DEBUG [...] 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:50 DEBUG 05:46:50 DEBUG "pod": "ds-idrepo-2" 05:46:50 DEBUG }, 05:46:50 DEBUG "instance": "10.142.0.43:9100" 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 1713761210, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 1713761210, 05:46:50 DEBUG "0" 05:46:50 DEBUG 1713761210, 05:46:50 DEBUG "0" 05:46:50 DEBUG ] 05:46:50 DEBUG "0.0005666666668086389" 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG } 05:46:50 DEBUG } 05:46:50 DEBUG } 05:46:50 DEBUG } 05:46:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761210" 05:46:50 INFO [http_cmd]: http status code OK 05:46:50 DEBUG --- status code --- 05:46:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:50 DEBUG --- http response --- 05:46:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:50 DEBUG ----- output ----- 05:46:50 DEBUG { 05:46:50 DEBUG "status": "success", 05:46:50 DEBUG "data": { 05:46:50 DEBUG "resultType": "vector", 05:46:50 DEBUG "result": [ 05:46:50 DEBUG { 05:46:50 DEBUG "metric": { 05:46:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG [...] 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 1713761210, 05:46:50 DEBUG "991.8695" 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG } 05:46:50 INFO [http_cmd]: http status code OK 05:46:50 DEBUG --- status code --- 05:46:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:50 DEBUG --- http response --- 05:46:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:50 DEBUG ----- output ----- 05:46:50 DEBUG { 05:46:50 DEBUG "status": "success", 05:46:50 DEBUG "data": { 05:46:50 DEBUG "resultType": "vector", 05:46:50 DEBUG "result": [ 05:46:50 DEBUG { 05:46:50 DEBUG "metric": { 05:46:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG [...] 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 1713761210, 05:46:50 DEBUG "362.79893711248894" 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG } 05:46:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761210" 05:46:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761210" 05:46:50 INFO [http_cmd]: http status code OK 05:46:50 DEBUG --- status code --- 05:46:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:50 DEBUG --- http response --- 05:46:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:50 DEBUG ----- output ----- 05:46:50 DEBUG { 05:46:50 DEBUG "status": "success", 05:46:50 DEBUG "data": { 05:46:50 DEBUG "resultType": "vector", 05:46:50 DEBUG "result": [ 05:46:50 DEBUG { 05:46:50 DEBUG "metric": { 05:46:50 DEBUG "instance": "10.142.0.107:9100" 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG [...] 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG "instance": "10.142.0.43:9100" 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 1713761210, 05:46:50 DEBUG "483089.1" 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG } 05:46:50 INFO [http_cmd]: http status code OK 05:46:50 DEBUG --- status code --- 05:46:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:50 DEBUG --- http response --- 05:46:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:50 DEBUG ----- output ----- 05:46:50 DEBUG { 05:46:50 DEBUG "status": "success", 05:46:50 DEBUG "data": { 05:46:50 DEBUG "resultType": "vector", 05:46:50 DEBUG "result": [ 05:46:50 DEBUG { 05:46:50 DEBUG "metric": { 05:46:50 DEBUG "instance": "10.142.0.107:9100" 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG [...] 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG 05:46:50 DEBUG "instance": "10.142.0.43:9100" 05:46:50 DEBUG }, 05:46:50 DEBUG "value": [ 05:46:50 DEBUG 1713761210, 05:46:50 DEBUG "0.0006666666666660604" 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG ] 05:46:50 DEBUG } 05:46:50 DEBUG } 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761211" 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "10196111360" 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761211" 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761211" 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761211" 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761211" 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761211" 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761211" 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:46:51 DEBUG }, 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:51 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:46:51 DEBUG }, 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "0" 05:46:51 DEBUG "0" 05:46:51 DEBUG ] 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "0" 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "pod": "ds-idrepo-0" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "pod": "ds-cts-2" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "0" 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "0" 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "0.8740848020132433" 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761211" 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761211" 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "instance": "10.142.0.107:9100" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "instance": "10.142.0.43:9100" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "476523.8" 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "0.8608686902861318" 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761211" 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "pod": "ds-cts-0" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "pod": "ds-cts-2" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "0" 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761211" 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "0" 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:46:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761211" 05:46:51 INFO [http_cmd]: http status code OK 05:46:51 DEBUG --- status code --- 05:46:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:51 DEBUG --- http response --- 05:46:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:51 DEBUG ----- output ----- 05:46:51 DEBUG { 05:46:51 DEBUG "status": "success", 05:46:51 DEBUG "data": { 05:46:51 DEBUG "resultType": "vector", 05:46:51 DEBUG "result": [ 05:46:51 DEBUG { 05:46:51 DEBUG "metric": { 05:46:51 DEBUG "pod": "ds-idrepo-0" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG [...] 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG 05:46:51 DEBUG "pod": "ds-idrepo-2" 05:46:51 DEBUG }, 05:46:51 DEBUG "value": [ 05:46:51 DEBUG 1713761211, 05:46:51 DEBUG "0" 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG ] 05:46:51 DEBUG } 05:46:51 DEBUG } 05:47:08 INFO 05:47:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:47:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:47:08 INFO [loop_until]: OK (rc = 0) 05:47:08 DEBUG --- stdout --- 05:47:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 177m 8790Mi am-767f6b67b5-qcqz2 224m 8839Mi am-767f6b67b5-vbltz 173m 8165Mi ds-cts-0 43m 5290Mi ds-cts-1 36m 11386Mi ds-cts-2 5946m 10629Mi ds-idrepo-0 14m 4010Mi ds-idrepo-1 12m 3992Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1243Mi lodemon-7f68c498d9-gbc5j 4m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 286m 9771Mi 05:47:08 DEBUG --- stderr --- 05:47:08 DEBUG 05:47:16 INFO 05:47:16 INFO [loop_until]: kubectl --namespace=xlou top node 05:47:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:47:16 INFO [loop_until]: OK (rc = 0) 05:47:16 DEBUG --- stdout --- 05:47:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6232Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 8113Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 238m 0% 11525Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 101m 0% 8528Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 5207Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 154m 0% 11728Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 3140Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 219m 0% 10420Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 7475Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 90m 0% 16559Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 87m 0% 8029Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 8133m 27% 13333Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 522m 3% 20141Mi 34% 05:47:16 DEBUG --- stderr --- 05:47:16 DEBUG 05:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761267" 05:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761267" 05:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761267" 05:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761267" 05:47:47 ERROR [http_cmd]: ERROR 05:47:47 INFO [http_cmd]: http status code OK 05:47:47 INFO [http_cmd]: http status code OK 05:47:47 INFO [http_cmd]: http status code OK 05:47:47 ERROR --- status code --- 05:47:47 DEBUG --- status code --- 05:47:47 DEBUG --- status code --- 05:47:47 DEBUG --- status code --- 05:47:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:47 ERROR --- http response --- 05:47:47 DEBUG --- http response --- 05:47:47 DEBUG --- http response --- 05:47:47 DEBUG --- http response --- 05:47:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:47:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:47 DEBUG ----- output ----- 05:47:47 DEBUG ----- output ----- 05:47:47 DEBUG { 05:47:47 DEBUG { 05:47:47 DEBUG "status": "success", 05:47:47 DEBUG "status": "success", 05:47:47 DEBUG "data": { 05:47:47 DEBUG "data": { 05:47:47 DEBUG "resultType": "vector", 05:47:47 DEBUG "resultType": "vector", 05:47:47 DEBUG "result": [ 05:47:47 DEBUG "result": [ 05:47:47 DEBUG { 05:47:47 DEBUG { 05:47:47 DEBUG "metric": { 05:47:47 DEBUG "metric": { 05:47:47 DEBUG "pod": "ds-idrepo-0" 05:47:47 DEBUG "pod": "ds-cts-0" 05:47:47 DEBUG }, 05:47:47 DEBUG }, 05:47:47 DEBUG "value": [ 05:47:47 DEBUG "value": [ 05:47:47 DEBUG 05:47:47 DEBUG 05:47:47 DEBUG 05:47:47 DEBUG 05:47:47 DEBUG 05:47:47 DEBUG 05:47:47 DEBUG [...] 05:47:47 DEBUG [...] 05:47:47 DEBUG 05:47:47 DEBUG 05:47:47 DEBUG 05:47:47 DEBUG 05:47:47 DEBUG 05:47:47 DEBUG 05:47:47 DEBUG "pod": "ds-idrepo-2" 05:47:47 DEBUG "pod": "ds-cts-2" 05:47:47 DEBUG }, 05:47:47 DEBUG }, 05:47:47 DEBUG "value": [ 05:47:47 DEBUG "value": [ 05:47:47 DEBUG 1713761267, 05:47:47 DEBUG 1713761267, 05:47:47 DEBUG "0" 05:47:47 DEBUG "0" 05:47:47 DEBUG ] 05:47:47 DEBUG ] 05:47:47 DEBUG } 05:47:47 DEBUG } 05:47:47 DEBUG ] 05:47:47 DEBUG ] 05:47:47 DEBUG } 05:47:47 DEBUG } 05:47:47 DEBUG } 05:47:47 DEBUG } 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761268" 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG { 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "data": { 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "result": [ 05:47:48 DEBUG { 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG "pod": "ds-idrepo-0" 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 1713761268, 05:47:48 DEBUG "186337938.740358" 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761269" 05:47:49 INFO [http_cmd]: http status code OK 05:47:49 DEBUG --- status code --- 05:47:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:49 DEBUG --- http response --- 05:47:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:49 DEBUG ----- output ----- 05:47:49 DEBUG { 05:47:49 DEBUG "status": "success", 05:47:49 DEBUG "data": { 05:47:49 DEBUG "resultType": "vector", 05:47:49 DEBUG "result": [ 05:47:49 DEBUG { 05:47:49 DEBUG "metric": { 05:47:49 DEBUG "instance": "10.142.0.107:9100" 05:47:49 DEBUG }, 05:47:49 DEBUG "value": [ 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG [...] 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG "instance": "10.142.0.43:9100" 05:47:49 DEBUG }, 05:47:49 DEBUG "value": [ 05:47:49 DEBUG 1713761269, 05:47:49 DEBUG "5832694350848" 05:47:49 DEBUG ] 05:47:49 DEBUG } 05:47:49 DEBUG ] 05:47:49 DEBUG } 05:47:49 DEBUG } 05:47:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761269" 05:47:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761269" 05:47:49 INFO [http_cmd]: http status code OK 05:47:49 DEBUG --- status code --- 05:47:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:49 DEBUG --- http response --- 05:47:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:49 DEBUG ----- output ----- 05:47:49 DEBUG { 05:47:49 DEBUG "status": "success", 05:47:49 DEBUG "data": { 05:47:49 DEBUG "resultType": "vector", 05:47:49 DEBUG "result": [ 05:47:49 DEBUG { 05:47:49 DEBUG "metric": { 05:47:49 DEBUG "pod": "ds-cts-0" 05:47:49 DEBUG }, 05:47:49 DEBUG "value": [ 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG [...] 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG "pod": "ds-cts-2" 05:47:49 DEBUG }, 05:47:49 DEBUG "value": [ 05:47:49 DEBUG 1713761269, 05:47:49 DEBUG "0" 05:47:49 DEBUG ] 05:47:49 DEBUG } 05:47:49 DEBUG ] 05:47:49 DEBUG } 05:47:49 DEBUG } 05:47:49 INFO [http_cmd]: http status code OK 05:47:49 DEBUG --- status code --- 05:47:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:49 DEBUG --- http response --- 05:47:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:49 DEBUG ----- output ----- 05:47:49 DEBUG { 05:47:49 DEBUG "status": "success", 05:47:49 DEBUG "data": { 05:47:49 DEBUG "resultType": "vector", 05:47:49 DEBUG "result": [ 05:47:49 DEBUG { 05:47:49 DEBUG "metric": { 05:47:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:49 DEBUG }, 05:47:49 DEBUG "value": [ 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG [...] 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG 05:47:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:49 DEBUG }, 05:47:49 DEBUG "value": [ 05:47:49 DEBUG 1713761269, 05:47:49 DEBUG "0" 05:47:49 DEBUG ] 05:47:49 DEBUG } 05:47:49 DEBUG ] 05:47:49 DEBUG } 05:47:49 DEBUG } 05:47:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761270" 05:47:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761270" 05:47:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761270" 05:47:50 INFO [http_cmd]: http status code OK 05:47:50 DEBUG --- status code --- 05:47:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:50 DEBUG --- http response --- 05:47:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:50 DEBUG ----- output ----- 05:47:50 DEBUG { 05:47:50 DEBUG "status": "success", 05:47:50 DEBUG "data": { 05:47:50 DEBUG "resultType": "vector", 05:47:50 DEBUG "result": [ 05:47:50 DEBUG { 05:47:50 DEBUG "metric": { 05:47:50 DEBUG "instance": "10.142.0.107:9100" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG [...] 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG "instance": "10.142.0.43:9100" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 1713761270, 05:47:50 DEBUG "0.0004666666668072139" 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG } 05:47:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761270" 05:47:50 INFO [http_cmd]: http status code OK 05:47:50 INFO [http_cmd]: http status code OK 05:47:50 DEBUG --- status code --- 05:47:50 DEBUG --- status code --- 05:47:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:50 DEBUG --- http response --- 05:47:50 DEBUG --- http response --- 05:47:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:50 DEBUG ----- output ----- 05:47:50 DEBUG { 05:47:50 DEBUG "status": "success", 05:47:50 DEBUG "data": { 05:47:50 DEBUG "resultType": "vector", 05:47:50 DEBUG "result": [ 05:47:50 DEBUG { 05:47:50 DEBUG "metric": { 05:47:50 DEBUG "pod": "ds-idrepo-0" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG [...] 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG "pod": "ds-idrepo-2" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:50 DEBUG 1713761270, 05:47:50 DEBUG ----- output ----- 05:47:50 DEBUG "0" 05:47:50 DEBUG { 05:47:50 DEBUG ] 05:47:50 DEBUG "status": "success", 05:47:50 DEBUG } 05:47:50 DEBUG "data": { 05:47:50 DEBUG ] 05:47:50 DEBUG "resultType": "vector", 05:47:50 DEBUG } 05:47:50 DEBUG "result": [ 05:47:50 DEBUG } 05:47:50 DEBUG { 05:47:50 DEBUG "metric": { 05:47:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG [...] 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 1713761270, 05:47:50 DEBUG "0" 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG } 05:47:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761270" 05:47:50 INFO [http_cmd]: http status code OK 05:47:50 DEBUG --- status code --- 05:47:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:50 DEBUG --- http response --- 05:47:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:50 DEBUG ----- output ----- 05:47:50 DEBUG { 05:47:50 DEBUG "status": "success", 05:47:50 DEBUG "data": { 05:47:50 DEBUG "resultType": "vector", 05:47:50 DEBUG "result": [ 05:47:50 DEBUG { 05:47:50 DEBUG "metric": { 05:47:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG [...] 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 1713761270, 05:47:50 DEBUG "991.914" 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG } 05:47:50 INFO [http_cmd]: http status code OK 05:47:50 DEBUG --- status code --- 05:47:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:50 DEBUG --- http response --- 05:47:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:47:50 DEBUG ----- output ----- 05:47:50 DEBUG { 05:47:50 DEBUG "status": "success", 05:47:50 DEBUG "data": { 05:47:50 DEBUG "resultType": "vector", 05:47:50 DEBUG "result": [ 05:47:50 DEBUG { 05:47:50 DEBUG "metric": { 05:47:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG [...] 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG "pod": "ds-idrepo-0" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 1713761270, 05:47:50 DEBUG "4754.586984198308" 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG } 05:47:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761270" 05:47:50 INFO [http_cmd]: http status code OK 05:47:50 DEBUG --- status code --- 05:47:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:50 DEBUG --- http response --- 05:47:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:50 DEBUG ----- output ----- 05:47:50 DEBUG { 05:47:50 DEBUG "status": "success", 05:47:50 DEBUG "data": { 05:47:50 DEBUG "resultType": "vector", 05:47:50 DEBUG "result": [ 05:47:50 DEBUG { 05:47:50 DEBUG "metric": { 05:47:50 DEBUG "instance": "10.142.0.107:9100" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG [...] 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG "instance": "10.142.0.43:9100" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 1713761270, 05:47:50 DEBUG "0.0003333333333330302" 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG } 05:47:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761270" 05:47:50 INFO [http_cmd]: http status code OK 05:47:50 DEBUG --- status code --- 05:47:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:50 DEBUG --- http response --- 05:47:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:50 DEBUG ----- output ----- 05:47:50 DEBUG { 05:47:50 DEBUG "status": "success", 05:47:50 DEBUG "data": { 05:47:50 DEBUG "resultType": "vector", 05:47:50 DEBUG "result": [ 05:47:50 DEBUG { 05:47:50 DEBUG "metric": { 05:47:50 DEBUG "instance": "10.142.0.107:9100" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG [...] 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG 05:47:50 DEBUG "instance": "10.142.0.43:9100" 05:47:50 DEBUG }, 05:47:50 DEBUG "value": [ 05:47:50 DEBUG 1713761270, 05:47:50 DEBUG "37949.63333333333" 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG ] 05:47:50 DEBUG } 05:47:50 DEBUG } 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761271" 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG [...] 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG "10629087232" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761271" 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761271" 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG [...] 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG "0.9972971622654102" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG [...] 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "pod": "ds-cts-0" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG "0" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761271" 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761271" 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761271" 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761271" 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG [...] 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG "0" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG [...] 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG "0" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761271" 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "data": { 05:47:51 DEBUG "pod": "ds-idrepo-0" 05:47:51 DEBUG }, 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "value": [ 05:47:51 DEBUG "result": [ 05:47:51 DEBUG 05:47:51 DEBUG { 05:47:51 DEBUG 05:47:51 DEBUG "metric": { 05:47:51 DEBUG 05:47:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:51 DEBUG [...] 05:47:51 DEBUG }, 05:47:51 DEBUG 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "pod": "ds-cts-2" 05:47:51 DEBUG 05:47:51 DEBUG }, 05:47:51 DEBUG [...] 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG 05:47:51 DEBUG "0" 05:47:51 DEBUG 05:47:51 DEBUG ] 05:47:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:51 DEBUG } 05:47:51 DEBUG }, 05:47:51 DEBUG ] 05:47:51 DEBUG "value": [ 05:47:51 DEBUG } 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG } 05:47:51 DEBUG "0" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "pod": "ds-cts-0" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG [...] 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "pod": "ds-cts-2" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG "0" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761271" 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761271" 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "instance": "10.142.0.107:9100" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG [...] 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "instance": "10.142.0.43:9100" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG "30656" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG [...] 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "pod": "ds-cts-0" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG "0.009343729481799618" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761271" 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG [...] 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG "0" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:47:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761271" 05:47:51 INFO [http_cmd]: http status code OK 05:47:51 DEBUG --- status code --- 05:47:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:51 DEBUG --- http response --- 05:47:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:51 DEBUG ----- output ----- 05:47:51 DEBUG { 05:47:51 DEBUG "status": "success", 05:47:51 DEBUG "data": { 05:47:51 DEBUG "resultType": "vector", 05:47:51 DEBUG "result": [ 05:47:51 DEBUG { 05:47:51 DEBUG "metric": { 05:47:51 DEBUG "pod": "ds-idrepo-0" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG [...] 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG 05:47:51 DEBUG "pod": "ds-idrepo-2" 05:47:51 DEBUG }, 05:47:51 DEBUG "value": [ 05:47:51 DEBUG 1713761271, 05:47:51 DEBUG "0" 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG ] 05:47:51 DEBUG } 05:47:51 DEBUG } 05:48:08 INFO 05:48:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:48:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:48:08 INFO [loop_until]: OK (rc = 0) 05:48:08 DEBUG --- stdout --- 05:48:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8789Mi am-767f6b67b5-qcqz2 7m 8838Mi am-767f6b67b5-vbltz 8m 8164Mi ds-cts-0 14m 5293Mi ds-cts-1 13m 11384Mi ds-cts-2 14m 10624Mi ds-idrepo-0 14m 4010Mi ds-idrepo-1 13m 3993Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1243Mi lodemon-7f68c498d9-gbc5j 6m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1027m 10083Mi 05:48:08 DEBUG --- stderr --- 05:48:08 DEBUG 05:48:16 INFO 05:48:16 INFO [loop_until]: kubectl --namespace=xlou top node 05:48:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:48:16 INFO [loop_until]: OK (rc = 0) 05:48:16 DEBUG --- stdout --- 05:48:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6231Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 8105Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 11520Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3345Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 70m 0% 8528Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5226Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 11721Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 76m 0% 3135Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 59m 0% 10406Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 7483Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 79m 0% 16541Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 91m 0% 8033Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 71m 0% 13343Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1104m 6% 20316Mi 34% 05:48:16 DEBUG --- stderr --- 05:48:16 DEBUG 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761328" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761328" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761328" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761328" 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 ERROR [http_cmd]: ERROR 05:48:48 DEBUG --- status code --- 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 ERROR --- status code --- 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- status code --- 05:48:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 ERROR --- http response --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "data": { 05:48:48 DEBUG "data": { 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "result": [ 05:48:48 DEBUG "result": [ 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "pod": "ds-cts-0" 05:48:48 DEBUG "pod": "ds-idrepo-0" 05:48:48 DEBUG }, 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG "pod": "ds-cts-2" 05:48:48 DEBUG "pod": "ds-idrepo-2" 05:48:48 DEBUG }, 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 1713761328, 05:48:48 DEBUG 1713761328, 05:48:48 DEBUG "0" 05:48:48 DEBUG "0" 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761328" 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG { 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "data": { 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "result": [ 05:48:48 DEBUG { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG "pod": "ds-cts-2" 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 1713761328, 05:48:48 DEBUG "0" 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761329" 05:48:49 INFO [http_cmd]: http status code OK 05:48:49 DEBUG --- status code --- 05:48:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:49 DEBUG --- http response --- 05:48:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:49 DEBUG ----- output ----- 05:48:49 DEBUG { 05:48:49 DEBUG "status": "success", 05:48:49 DEBUG "data": { 05:48:49 DEBUG "resultType": "vector", 05:48:49 DEBUG "result": [ 05:48:49 DEBUG { 05:48:49 DEBUG "metric": { 05:48:49 DEBUG "instance": "10.142.0.107:9100" 05:48:49 DEBUG }, 05:48:49 DEBUG "value": [ 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG [...] 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG "instance": "10.142.0.43:9100" 05:48:49 DEBUG }, 05:48:49 DEBUG "value": [ 05:48:49 DEBUG 1713761329, 05:48:49 DEBUG "5832695102464" 05:48:49 DEBUG ] 05:48:49 DEBUG } 05:48:49 DEBUG ] 05:48:49 DEBUG } 05:48:49 DEBUG } 05:48:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761329" 05:48:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761329" 05:48:49 INFO [http_cmd]: http status code OK 05:48:49 DEBUG --- status code --- 05:48:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:49 DEBUG --- http response --- 05:48:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:49 DEBUG ----- output ----- 05:48:49 DEBUG { 05:48:49 DEBUG "status": "success", 05:48:49 DEBUG "data": { 05:48:49 DEBUG "resultType": "vector", 05:48:49 DEBUG "result": [ 05:48:49 DEBUG { 05:48:49 DEBUG "metric": { 05:48:49 DEBUG "pod": "ds-cts-0" 05:48:49 DEBUG }, 05:48:49 DEBUG "value": [ 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG [...] 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG "pod": "ds-cts-2" 05:48:49 DEBUG }, 05:48:49 DEBUG "value": [ 05:48:49 DEBUG 1713761329, 05:48:49 DEBUG "0" 05:48:49 DEBUG ] 05:48:49 DEBUG } 05:48:49 DEBUG ] 05:48:49 DEBUG } 05:48:49 DEBUG } 05:48:49 INFO [http_cmd]: http status code OK 05:48:49 DEBUG --- status code --- 05:48:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:49 DEBUG --- http response --- 05:48:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:49 DEBUG ----- output ----- 05:48:49 DEBUG { 05:48:49 DEBUG "status": "success", 05:48:49 DEBUG "data": { 05:48:49 DEBUG "resultType": "vector", 05:48:49 DEBUG "result": [ 05:48:49 DEBUG { 05:48:49 DEBUG "metric": { 05:48:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:49 DEBUG }, 05:48:49 DEBUG "value": [ 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG [...] 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG 05:48:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:49 DEBUG }, 05:48:49 DEBUG "value": [ 05:48:49 DEBUG 1713761329, 05:48:49 DEBUG "0" 05:48:49 DEBUG ] 05:48:49 DEBUG } 05:48:49 DEBUG ] 05:48:49 DEBUG } 05:48:49 DEBUG } 05:48:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761330" 05:48:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761330" 05:48:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761330" 05:48:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761330" 05:48:50 INFO [http_cmd]: http status code OK 05:48:50 DEBUG --- status code --- 05:48:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:50 DEBUG --- http response --- 05:48:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:50 DEBUG ----- output ----- 05:48:50 DEBUG { 05:48:50 DEBUG "status": "success", 05:48:50 DEBUG "data": { 05:48:50 DEBUG "resultType": "vector", 05:48:50 DEBUG "result": [ 05:48:50 DEBUG { 05:48:50 DEBUG "metric": { 05:48:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:50 DEBUG }, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG [...] 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:50 DEBUG }, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG 1713761330, 05:48:50 DEBUG "0" 05:48:50 DEBUG ] 05:48:50 DEBUG } 05:48:50 DEBUG ] 05:48:50 DEBUG } 05:48:50 DEBUG } 05:48:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761330" 05:48:50 INFO [http_cmd]: http status code OK 05:48:50 INFO [http_cmd]: http status code OK 05:48:50 DEBUG --- status code --- 05:48:50 DEBUG --- status code --- 05:48:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:50 DEBUG --- http response --- 05:48:50 DEBUG --- http response --- 05:48:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:50 DEBUG ----- output ----- 05:48:50 DEBUG ----- output ----- 05:48:50 DEBUG { 05:48:50 DEBUG { 05:48:50 DEBUG "status": "success", 05:48:50 DEBUG "status": "success", 05:48:50 DEBUG "data": { 05:48:50 DEBUG "data": { 05:48:50 DEBUG "resultType": "vector", 05:48:50 DEBUG "resultType": "vector", 05:48:50 DEBUG "result": [ 05:48:50 DEBUG "result": [ 05:48:50 DEBUG { 05:48:50 INFO [http_cmd]: http status code OK 05:48:50 DEBUG { 05:48:50 DEBUG "metric": { 05:48:50 DEBUG --- status code --- 05:48:50 DEBUG "metric": { 05:48:50 DEBUG "pod": "ds-idrepo-0" 05:48:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:50 DEBUG }, 05:48:50 DEBUG --- http response --- 05:48:50 DEBUG }, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG "value": [ 05:48:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG ----- output ----- 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG { 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG "status": "success", 05:48:50 DEBUG [...] 05:48:50 DEBUG [...] 05:48:50 DEBUG "data": { 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG "resultType": "vector", 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG "result": [ 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG { 05:48:50 DEBUG "pod": "ds-idrepo-2" 05:48:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:50 DEBUG "metric": { 05:48:50 DEBUG }, 05:48:50 DEBUG }, 05:48:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:48:50 DEBUG "value": [ 05:48:50 DEBUG "value": [ 05:48:50 DEBUG }, 05:48:50 DEBUG 1713761330, 05:48:50 DEBUG 1713761330, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG "0" 05:48:50 DEBUG "991.9145" 05:48:50 DEBUG 05:48:50 DEBUG ] 05:48:50 DEBUG ] 05:48:50 DEBUG 05:48:50 DEBUG } 05:48:50 DEBUG } 05:48:50 DEBUG 05:48:50 DEBUG ] 05:48:50 DEBUG ] 05:48:50 DEBUG [...] 05:48:50 DEBUG } 05:48:50 DEBUG } 05:48:50 DEBUG 05:48:50 DEBUG } 05:48:50 DEBUG } 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:48:50 DEBUG }, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG 1713761330, 05:48:50 DEBUG "0" 05:48:50 DEBUG ] 05:48:50 DEBUG } 05:48:50 DEBUG ] 05:48:50 DEBUG } 05:48:50 DEBUG } 05:48:50 INFO [http_cmd]: http status code OK 05:48:50 DEBUG --- status code --- 05:48:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:50 DEBUG --- http response --- 05:48:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:50 DEBUG ----- output ----- 05:48:50 DEBUG { 05:48:50 DEBUG "status": "success", 05:48:50 DEBUG "data": { 05:48:50 DEBUG "resultType": "vector", 05:48:50 DEBUG "result": [ 05:48:50 DEBUG { 05:48:50 DEBUG "metric": { 05:48:50 DEBUG "instance": "10.142.0.107:9100" 05:48:50 DEBUG }, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG [...] 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG "instance": "10.142.0.43:9100" 05:48:50 DEBUG }, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG 1713761330, 05:48:50 DEBUG "0.0005333333332335617" 05:48:50 DEBUG ] 05:48:50 DEBUG } 05:48:50 DEBUG ] 05:48:50 DEBUG } 05:48:50 DEBUG } 05:48:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761330" 05:48:50 INFO [http_cmd]: http status code OK 05:48:50 DEBUG --- status code --- 05:48:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:50 DEBUG --- http response --- 05:48:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:50 DEBUG ----- output ----- 05:48:50 DEBUG { 05:48:50 DEBUG "status": "success", 05:48:50 DEBUG "data": { 05:48:50 DEBUG "resultType": "vector", 05:48:50 DEBUG "result": [ 05:48:50 DEBUG { 05:48:50 DEBUG "metric": { 05:48:50 DEBUG "instance": "10.142.0.107:9100" 05:48:50 DEBUG }, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG [...] 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG "instance": "10.142.0.43:9100" 05:48:50 DEBUG }, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG 1713761330, 05:48:50 DEBUG "0.0003333333333330302" 05:48:50 DEBUG ] 05:48:50 DEBUG } 05:48:50 DEBUG ] 05:48:50 DEBUG } 05:48:50 DEBUG } 05:48:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761330" 05:48:50 INFO [http_cmd]: http status code OK 05:48:50 DEBUG --- status code --- 05:48:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:50 DEBUG --- http response --- 05:48:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:50 DEBUG ----- output ----- 05:48:50 DEBUG { 05:48:50 DEBUG "status": "success", 05:48:50 DEBUG "data": { 05:48:50 DEBUG "resultType": "vector", 05:48:50 DEBUG "result": [ 05:48:50 DEBUG { 05:48:50 DEBUG "metric": { 05:48:50 DEBUG "instance": "10.142.0.107:9100" 05:48:50 DEBUG }, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG [...] 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG 05:48:50 DEBUG "instance": "10.142.0.43:9100" 05:48:50 DEBUG }, 05:48:50 DEBUG "value": [ 05:48:50 DEBUG 1713761330, 05:48:50 DEBUG "38065.5" 05:48:50 DEBUG ] 05:48:50 DEBUG } 05:48:50 DEBUG ] 05:48:50 DEBUG } 05:48:50 DEBUG } 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761331" 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "10778611712" 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761331" 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "1204580.3704847675" 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761331" 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "1.0168378414648924" 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761331" 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "instance": "10.142.0.107:9100" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "instance": "10.142.0.43:9100" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "31276.299999999996" 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761331" 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761331" 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761331" 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761331" 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "0" 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "0" 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "0" 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761331" 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761331" 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "pod": "ds-idrepo-0" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "pod": "ds-cts-2" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "0" 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "pod": "ds-cts-0" 05:48:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:48:51 DEBUG }, 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "pod": "ds-cts-2" 05:48:51 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:48:51 DEBUG }, 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "0" 05:48:51 DEBUG "1.9811336688828527" 05:48:51 DEBUG ] 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761331" 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "0" 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:48:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761331" 05:48:51 INFO [http_cmd]: http status code OK 05:48:51 DEBUG --- status code --- 05:48:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:51 DEBUG --- http response --- 05:48:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:51 DEBUG ----- output ----- 05:48:51 DEBUG { 05:48:51 DEBUG "status": "success", 05:48:51 DEBUG "data": { 05:48:51 DEBUG "resultType": "vector", 05:48:51 DEBUG "result": [ 05:48:51 DEBUG { 05:48:51 DEBUG "metric": { 05:48:51 DEBUG "pod": "ds-idrepo-0" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG [...] 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG 05:48:51 DEBUG "pod": "ds-idrepo-2" 05:48:51 DEBUG }, 05:48:51 DEBUG "value": [ 05:48:51 DEBUG 1713761331, 05:48:51 DEBUG "0" 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG ] 05:48:51 DEBUG } 05:48:51 DEBUG } 05:49:08 INFO 05:49:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:49:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:49:08 INFO [loop_until]: OK (rc = 0) 05:49:08 DEBUG --- stdout --- 05:49:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8789Mi am-767f6b67b5-qcqz2 7m 8838Mi am-767f6b67b5-vbltz 7m 8164Mi ds-cts-0 13m 5290Mi ds-cts-1 13m 11385Mi ds-cts-2 13m 10624Mi ds-idrepo-0 14m 4010Mi ds-idrepo-1 11m 3995Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1182Mi idm-b85f9d978-wnhxb 3m 1243Mi lodemon-7f68c498d9-gbc5j 3m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1028m 10206Mi 05:49:08 DEBUG --- stderr --- 05:49:08 DEBUG 05:49:16 INFO 05:49:16 INFO [loop_until]: kubectl --namespace=xlou top node 05:49:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:49:16 INFO [loop_until]: OK (rc = 0) 05:49:16 DEBUG --- stdout --- 05:49:16 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6226Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 8108Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 58m 0% 11518Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3354Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 8529Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 137m 0% 5215Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 11716Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 3128Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 58m 0% 10392Mi 9% gke-xlou-cdm-ds-0f601751-j488 70m 0% 7481Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 75m 0% 16544Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 91m 0% 8032Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 13338Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1079m 6% 20518Mi 34% 05:49:16 DEBUG --- stderr --- 05:49:16 DEBUG 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761388" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761388" 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761388" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761388" 05:49:48 ERROR [http_cmd]: ERROR 05:49:48 ERROR --- status code --- 05:49:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:49:48 ERROR --- http response --- 05:49:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:49:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "data": { 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "result": [ 05:49:48 DEBUG { 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "pod": "ds-cts-0" 05:49:48 DEBUG }, 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "pod": "ds-cts-2" 05:49:48 DEBUG }, 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 1713761388, 05:49:48 DEBUG "0" 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "data": { 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "result": [ 05:49:48 DEBUG { 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "pod": "ds-idrepo-0" 05:49:48 DEBUG }, 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "pod": "ds-idrepo-2" 05:49:48 DEBUG }, 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 1713761388, 05:49:48 DEBUG "0" 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761388" 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "data": { 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "result": [ 05:49:48 DEBUG { 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:49:48 DEBUG }, 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:49:48 DEBUG }, 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 1713761388, 05:49:48 DEBUG "2879777.4026768208" 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761389" 05:49:49 INFO [http_cmd]: http status code OK 05:49:49 DEBUG --- status code --- 05:49:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:49 DEBUG --- http response --- 05:49:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:49 DEBUG ----- output ----- 05:49:49 DEBUG { 05:49:49 DEBUG "status": "success", 05:49:49 DEBUG "data": { 05:49:49 DEBUG "resultType": "vector", 05:49:49 DEBUG "result": [ 05:49:49 DEBUG { 05:49:49 DEBUG "metric": { 05:49:49 DEBUG "instance": "10.142.0.107:9100" 05:49:49 DEBUG }, 05:49:49 DEBUG "value": [ 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG [...] 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG "instance": "10.142.0.43:9100" 05:49:49 DEBUG }, 05:49:49 DEBUG "value": [ 05:49:49 DEBUG 1713761389, 05:49:49 DEBUG "5832695909376" 05:49:49 DEBUG ] 05:49:49 DEBUG } 05:49:49 DEBUG ] 05:49:49 DEBUG } 05:49:49 DEBUG } 05:49:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761389" 05:49:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761389" 05:49:49 INFO [http_cmd]: http status code OK 05:49:49 DEBUG --- status code --- 05:49:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:49 DEBUG --- http response --- 05:49:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:49 DEBUG ----- output ----- 05:49:49 DEBUG { 05:49:49 DEBUG "status": "success", 05:49:49 DEBUG "data": { 05:49:49 DEBUG "resultType": "vector", 05:49:49 DEBUG "result": [ 05:49:49 DEBUG { 05:49:49 DEBUG "metric": { 05:49:49 DEBUG "pod": "ds-cts-0" 05:49:49 DEBUG }, 05:49:49 DEBUG "value": [ 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG [...] 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG "pod": "ds-cts-2" 05:49:49 DEBUG }, 05:49:49 DEBUG "value": [ 05:49:49 DEBUG 1713761389, 05:49:49 DEBUG "0" 05:49:49 DEBUG ] 05:49:49 DEBUG } 05:49:49 DEBUG ] 05:49:49 DEBUG } 05:49:49 DEBUG } 05:49:49 INFO [http_cmd]: http status code OK 05:49:49 DEBUG --- status code --- 05:49:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:49 DEBUG --- http response --- 05:49:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:49 DEBUG ----- output ----- 05:49:49 DEBUG { 05:49:49 DEBUG "status": "success", 05:49:49 DEBUG "data": { 05:49:49 DEBUG "resultType": "vector", 05:49:49 DEBUG "result": [ 05:49:49 DEBUG { 05:49:49 DEBUG "metric": { 05:49:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:49 DEBUG }, 05:49:49 DEBUG "value": [ 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG [...] 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG 05:49:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:49 DEBUG }, 05:49:49 DEBUG "value": [ 05:49:49 DEBUG 1713761389, 05:49:49 DEBUG "0" 05:49:49 DEBUG ] 05:49:49 DEBUG } 05:49:49 DEBUG ] 05:49:49 DEBUG } 05:49:49 DEBUG } 05:49:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761390" 05:49:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761390" 05:49:50 INFO [http_cmd]: http status code OK 05:49:50 DEBUG --- status code --- 05:49:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:50 DEBUG --- http response --- 05:49:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:50 DEBUG ----- output ----- 05:49:50 DEBUG { 05:49:50 DEBUG "status": "success", 05:49:50 DEBUG "data": { 05:49:50 DEBUG "resultType": "vector", 05:49:50 DEBUG "result": [ 05:49:50 DEBUG { 05:49:50 DEBUG "metric": { 05:49:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG [...] 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG "pod": "ds-cts-0" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 1713761390, 05:49:50 DEBUG "834.4141513597936" 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG } 05:49:50 INFO [http_cmd]: http status code OK 05:49:50 DEBUG --- status code --- 05:49:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:50 DEBUG --- http response --- 05:49:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:50 DEBUG ----- output ----- 05:49:50 DEBUG { 05:49:50 DEBUG "status": "success", 05:49:50 DEBUG "data": { 05:49:50 DEBUG "resultType": "vector", 05:49:50 DEBUG "result": [ 05:49:50 DEBUG { 05:49:50 DEBUG "metric": { 05:49:50 DEBUG "pod": "ds-idrepo-0" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG [...] 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG "pod": "ds-idrepo-2" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 1713761390, 05:49:50 DEBUG "0" 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG } 05:49:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761390" 05:49:50 INFO [http_cmd]: http status code OK 05:49:50 DEBUG --- status code --- 05:49:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:50 DEBUG --- http response --- 05:49:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:50 DEBUG ----- output ----- 05:49:50 DEBUG { 05:49:50 DEBUG "status": "success", 05:49:50 DEBUG "data": { 05:49:50 DEBUG "resultType": "vector", 05:49:50 DEBUG "result": [ 05:49:50 DEBUG { 05:49:50 DEBUG "metric": { 05:49:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG [...] 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 1713761390, 05:49:50 DEBUG "0" 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG } 05:49:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761390" 05:49:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761390" 05:49:50 INFO [http_cmd]: http status code OK 05:49:50 DEBUG --- status code --- 05:49:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:50 DEBUG --- http response --- 05:49:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:50 DEBUG ----- output ----- 05:49:50 DEBUG { 05:49:50 DEBUG "status": "success", 05:49:50 DEBUG "data": { 05:49:50 DEBUG "resultType": "vector", 05:49:50 DEBUG "result": [ 05:49:50 DEBUG { 05:49:50 DEBUG "metric": { 05:49:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG [...] 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 1713761390, 05:49:50 DEBUG "991.915" 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG } 05:49:50 INFO [http_cmd]: http status code OK 05:49:50 DEBUG --- status code --- 05:49:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:50 DEBUG --- http response --- 05:49:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:50 DEBUG ----- output ----- 05:49:50 DEBUG { 05:49:50 DEBUG "status": "success", 05:49:50 DEBUG "data": { 05:49:50 DEBUG "resultType": "vector", 05:49:50 DEBUG "result": [ 05:49:50 DEBUG { 05:49:50 DEBUG "metric": { 05:49:50 DEBUG "instance": "10.142.0.107:9100" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG [...] 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG "instance": "10.142.0.43:9100" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 1713761390, 05:49:50 DEBUG "0.0005666666666930572" 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG } 05:49:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761390" 05:49:50 INFO [http_cmd]: http status code OK 05:49:50 DEBUG --- status code --- 05:49:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:50 DEBUG --- http response --- 05:49:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:50 DEBUG ----- output ----- 05:49:50 DEBUG { 05:49:50 DEBUG "status": "success", 05:49:50 DEBUG "data": { 05:49:50 DEBUG "resultType": "vector", 05:49:50 DEBUG "result": [ 05:49:50 DEBUG { 05:49:50 DEBUG "metric": { 05:49:50 DEBUG "instance": "10.142.0.107:9100" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG [...] 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG "instance": "10.142.0.43:9100" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 1713761390, 05:49:50 DEBUG "0" 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG } 05:49:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761390" 05:49:50 INFO [http_cmd]: http status code OK 05:49:50 DEBUG --- status code --- 05:49:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:50 DEBUG --- http response --- 05:49:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:50 DEBUG ----- output ----- 05:49:50 DEBUG { 05:49:50 DEBUG "status": "success", 05:49:50 DEBUG "data": { 05:49:50 DEBUG "resultType": "vector", 05:49:50 DEBUG "result": [ 05:49:50 DEBUG { 05:49:50 DEBUG "metric": { 05:49:50 DEBUG "instance": "10.142.0.107:9100" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG [...] 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG 05:49:50 DEBUG "instance": "10.142.0.43:9100" 05:49:50 DEBUG }, 05:49:50 DEBUG "value": [ 05:49:50 DEBUG 1713761390, 05:49:50 DEBUG "38385.166666666664" 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG ] 05:49:50 DEBUG } 05:49:50 DEBUG } 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761391" 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "10939932672" 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761391" 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG "pod": "ds-cts-0" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "0" 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761391" 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761391" 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761391" 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761391" 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "1.0360442343023994" 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:51 DEBUG }, 05:49:51 DEBUG "instance": "10.142.0.107:9100" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:51 DEBUG 05:49:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:51 DEBUG }, 05:49:51 DEBUG "instance": "10.142.0.43:9100" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "0" 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "0" 05:49:51 DEBUG ] 05:49:51 DEBUG "36457.166666666664" 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761391" 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG "pod": "ds-cts-0" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "0.012915628287023332" 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761391" 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761391" 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "0" 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761391" 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "pod": "ds-idrepo-0" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG "pod": "ds-cts-2" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "0" 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "pod": "ds-cts-0" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG "pod": "ds-cts-2" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "0" 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761391" 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "pod": "ds-idrepo-0" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG "pod": "ds-idrepo-2" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "0" 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG } 05:49:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761391" 05:49:51 INFO [http_cmd]: http status code OK 05:49:51 DEBUG --- status code --- 05:49:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:51 DEBUG --- http response --- 05:49:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:51 DEBUG ----- output ----- 05:49:51 DEBUG { 05:49:51 DEBUG "status": "success", 05:49:51 DEBUG "data": { 05:49:51 DEBUG "resultType": "vector", 05:49:51 DEBUG "result": [ 05:49:51 DEBUG { 05:49:51 DEBUG "metric": { 05:49:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG [...] 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG 05:49:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:51 DEBUG }, 05:49:51 DEBUG "value": [ 05:49:51 DEBUG 1713761391, 05:49:51 DEBUG "0" 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG ] 05:49:51 DEBUG } 05:49:51 DEBUG } 05:50:08 INFO 05:50:08 INFO [loop_until]: kubectl --namespace=xlou top pods 05:50:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:50:08 INFO [loop_until]: OK (rc = 0) 05:50:08 DEBUG --- stdout --- 05:50:08 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8789Mi am-767f6b67b5-qcqz2 7m 8839Mi am-767f6b67b5-vbltz 8m 8165Mi ds-cts-0 12m 5290Mi ds-cts-1 14m 11385Mi ds-cts-2 13m 10625Mi ds-idrepo-0 13m 4009Mi ds-idrepo-1 13m 3993Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1243Mi lodemon-7f68c498d9-gbc5j 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-68c5d5b884-p97w5 997m 10429Mi 05:50:08 DEBUG --- stderr --- 05:50:08 DEBUG 05:50:16 INFO 05:50:16 INFO [loop_until]: kubectl --namespace=xlou top node 05:50:16 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:50:17 INFO [loop_until]: OK (rc = 0) 05:50:17 DEBUG --- stdout --- 05:50:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6235Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 8115Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 11518Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 71m 0% 8521Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5208Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 11716Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 76m 0% 3131Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 60m 0% 10391Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 7490Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 79m 0% 16547Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 90m 0% 8046Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 67m 0% 13328Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1080m 6% 20640Mi 35% 05:50:17 DEBUG --- stderr --- 05:50:17 DEBUG 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761448" 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761448" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761448" 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "data": { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "result": [ 05:50:48 DEBUG { 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "pod": "ds-cts-0" 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "pod": "ds-cts-2" 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 1713761448, 05:50:48 DEBUG "0" 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761448" 05:50:48 ERROR [http_cmd]: ERROR 05:50:48 ERROR --- status code --- 05:50:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:50:48 ERROR --- http response --- 05:50:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:50:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "data": { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "result": [ 05:50:48 DEBUG { 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "pod": "ds-idrepo-0" 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "pod": "ds-idrepo-2" 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 1713761448, 05:50:48 DEBUG "0" 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761448" 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "data": { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "result": [ 05:50:48 DEBUG { 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "pod": "ds-cts-2" 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 1713761448, 05:50:48 DEBUG "0" 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761449" 05:50:49 INFO [http_cmd]: http status code OK 05:50:49 DEBUG --- status code --- 05:50:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:49 DEBUG --- http response --- 05:50:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:49 DEBUG ----- output ----- 05:50:49 DEBUG { 05:50:49 DEBUG "status": "success", 05:50:49 DEBUG "data": { 05:50:49 DEBUG "resultType": "vector", 05:50:49 DEBUG "result": [ 05:50:49 DEBUG { 05:50:49 DEBUG "metric": { 05:50:49 DEBUG "instance": "10.142.0.107:9100" 05:50:49 DEBUG }, 05:50:49 DEBUG "value": [ 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG [...] 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG "instance": "10.142.0.43:9100" 05:50:49 DEBUG }, 05:50:49 DEBUG "value": [ 05:50:49 DEBUG 1713761449, 05:50:49 DEBUG "5832696648704" 05:50:49 DEBUG ] 05:50:49 DEBUG } 05:50:49 DEBUG ] 05:50:49 DEBUG } 05:50:49 DEBUG } 05:50:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761449" 05:50:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761449" 05:50:49 INFO [http_cmd]: http status code OK 05:50:49 DEBUG --- status code --- 05:50:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:49 DEBUG --- http response --- 05:50:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:49 DEBUG ----- output ----- 05:50:49 DEBUG { 05:50:49 DEBUG "status": "success", 05:50:49 DEBUG "data": { 05:50:49 DEBUG "resultType": "vector", 05:50:49 DEBUG "result": [ 05:50:49 DEBUG { 05:50:49 DEBUG "metric": { 05:50:49 DEBUG "pod": "ds-cts-0" 05:50:49 DEBUG }, 05:50:49 DEBUG "value": [ 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG [...] 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG "pod": "ds-cts-2" 05:50:49 DEBUG }, 05:50:49 DEBUG "value": [ 05:50:49 DEBUG 1713761449, 05:50:49 DEBUG "0" 05:50:49 DEBUG ] 05:50:49 DEBUG } 05:50:49 DEBUG ] 05:50:49 DEBUG } 05:50:49 DEBUG } 05:50:49 INFO [http_cmd]: http status code OK 05:50:49 DEBUG --- status code --- 05:50:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:49 DEBUG --- http response --- 05:50:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:49 DEBUG ----- output ----- 05:50:49 DEBUG { 05:50:49 DEBUG "status": "success", 05:50:49 DEBUG "data": { 05:50:49 DEBUG "resultType": "vector", 05:50:49 DEBUG "result": [ 05:50:49 DEBUG { 05:50:49 DEBUG "metric": { 05:50:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:49 DEBUG }, 05:50:49 DEBUG "value": [ 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG [...] 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG 05:50:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:49 DEBUG }, 05:50:49 DEBUG "value": [ 05:50:49 DEBUG 1713761449, 05:50:49 DEBUG "0" 05:50:49 DEBUG ] 05:50:49 DEBUG } 05:50:49 DEBUG ] 05:50:49 DEBUG } 05:50:49 DEBUG } 05:50:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761450" 05:50:50 INFO [http_cmd]: http status code OK 05:50:50 DEBUG --- status code --- 05:50:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:50 DEBUG --- http response --- 05:50:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:50 DEBUG ----- output ----- 05:50:50 DEBUG { 05:50:50 DEBUG "status": "success", 05:50:50 DEBUG "data": { 05:50:50 DEBUG "resultType": "vector", 05:50:50 DEBUG "result": [ 05:50:50 DEBUG { 05:50:50 DEBUG "metric": { 05:50:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:50:50 DEBUG }, 05:50:50 DEBUG "value": [ 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG [...] 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:50 DEBUG }, 05:50:50 DEBUG "value": [ 05:50:50 DEBUG 1713761450, 05:50:50 DEBUG "550.0016784719191" 05:50:50 DEBUG ] 05:50:50 DEBUG } 05:50:50 DEBUG ] 05:50:50 DEBUG } 05:50:50 DEBUG } 05:50:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761450" 05:50:50 INFO [http_cmd]: http status code OK 05:50:50 DEBUG --- status code --- 05:50:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:50 DEBUG --- http response --- 05:50:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:50 DEBUG ----- output ----- 05:50:50 DEBUG { 05:50:50 DEBUG "status": "success", 05:50:50 DEBUG "data": { 05:50:50 DEBUG "resultType": "vector", 05:50:50 DEBUG "result": [ 05:50:50 DEBUG { 05:50:50 DEBUG "metric": { 05:50:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:50 DEBUG }, 05:50:50 DEBUG "value": [ 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG [...] 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:50 DEBUG }, 05:50:50 DEBUG "value": [ 05:50:50 DEBUG 1713761450, 05:50:50 DEBUG "0" 05:50:50 DEBUG ] 05:50:50 DEBUG } 05:50:50 DEBUG ] 05:50:50 DEBUG } 05:50:50 DEBUG } 05:50:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761450" 05:50:50 INFO [http_cmd]: http status code OK 05:50:50 DEBUG --- status code --- 05:50:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:50 DEBUG --- http response --- 05:50:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:50 DEBUG ----- output ----- 05:50:50 DEBUG { 05:50:50 DEBUG "status": "success", 05:50:50 DEBUG "data": { 05:50:50 DEBUG "resultType": "vector", 05:50:50 DEBUG "result": [ 05:50:50 DEBUG { 05:50:50 DEBUG "metric": { 05:50:50 DEBUG "pod": "ds-idrepo-0" 05:50:50 DEBUG }, 05:50:50 DEBUG "value": [ 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG [...] 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG "pod": "ds-idrepo-2" 05:50:50 DEBUG }, 05:50:50 DEBUG "value": [ 05:50:50 DEBUG 1713761450, 05:50:50 DEBUG "0" 05:50:50 DEBUG ] 05:50:50 DEBUG } 05:50:50 DEBUG ] 05:50:50 DEBUG } 05:50:50 DEBUG } 05:50:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761450" 05:50:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761450" 05:50:50 INFO [http_cmd]: http status code OK 05:50:50 DEBUG --- status code --- 05:50:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:50 DEBUG --- http response --- 05:50:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:50 DEBUG ----- output ----- 05:50:50 DEBUG { 05:50:50 DEBUG "status": "success", 05:50:50 DEBUG "data": { 05:50:50 DEBUG "resultType": "vector", 05:50:50 DEBUG "result": [ 05:50:50 DEBUG { 05:50:50 DEBUG "metric": { 05:50:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:50 DEBUG }, 05:50:50 DEBUG "value": [ 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG [...] 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:50 DEBUG }, 05:50:50 DEBUG "value": [ 05:50:50 DEBUG 1713761450, 05:50:50 DEBUG "991.915" 05:50:50 DEBUG ] 05:50:50 DEBUG } 05:50:50 DEBUG ] 05:50:50 DEBUG } 05:50:50 DEBUG } 05:50:50 INFO [http_cmd]: http status code OK 05:50:50 DEBUG --- status code --- 05:50:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:50 DEBUG --- http response --- 05:50:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:50 DEBUG ----- output ----- 05:50:50 DEBUG { 05:50:50 DEBUG "status": "success", 05:50:50 DEBUG "data": { 05:50:50 DEBUG "resultType": "vector", 05:50:50 DEBUG "result": [ 05:50:50 DEBUG { 05:50:50 DEBUG "metric": { 05:50:50 DEBUG "instance": "10.142.0.107:9100" 05:50:50 DEBUG }, 05:50:50 DEBUG "value": [ 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG [...] 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG 05:50:50 DEBUG "instance": "10.142.0.43:9100" 05:50:50 DEBUG }, 05:50:50 DEBUG "value": [ 05:50:50 DEBUG 1713761450, 05:50:50 DEBUG "0.0004666666666935271" 05:50:50 DEBUG ] 05:50:50 DEBUG } 05:50:50 DEBUG ] 05:50:50 DEBUG } 05:50:50 DEBUG } 05:50:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761450" 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761451" 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "instance": "10.142.0.107:9100" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "instance": "10.142.0.43:9100" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761450, 05:50:51 DEBUG "38355.3" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "instance": "10.142.0.107:9100" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "instance": "10.142.0.43:9100" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "0.0003333333333330302" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761451" 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "11078787072" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761451" 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "0" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761451" 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761451" 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "0" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "1.033991134646711" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761451" 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761451" 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG "result": [ 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG { 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:51 DEBUG }, 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG "value": [ 05:50:51 DEBUG { 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "0" 05:50:51 DEBUG "data": { 05:50:51 DEBUG ] 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG } 05:50:51 DEBUG "result": [ 05:50:51 DEBUG ] 05:50:51 DEBUG { 05:50:51 DEBUG } 05:50:51 DEBUG "metric": { 05:50:51 DEBUG } 05:50:51 DEBUG "instance": "10.142.0.107:9100" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "instance": "10.142.0.43:9100" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "31365.266666666663" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761451" 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761451" 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "pod": "ds-cts-0" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "pod": "ds-cts-2" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "0" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "0.005892678503628772" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761451" 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761451" 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:51 DEBUG "pod": "ds-idrepo-0" 05:50:51 DEBUG }, 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:51 DEBUG "pod": "ds-cts-2" 05:50:51 DEBUG }, 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "0" 05:50:51 DEBUG "0" 05:50:51 DEBUG ] 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761451" 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "pod": "ds-idrepo-0" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "pod": "ds-idrepo-2" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "0" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:50:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761451" 05:50:51 INFO [http_cmd]: http status code OK 05:50:51 DEBUG --- status code --- 05:50:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:51 DEBUG --- http response --- 05:50:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:51 DEBUG ----- output ----- 05:50:51 DEBUG { 05:50:51 DEBUG "status": "success", 05:50:51 DEBUG "data": { 05:50:51 DEBUG "resultType": "vector", 05:50:51 DEBUG "result": [ 05:50:51 DEBUG { 05:50:51 DEBUG "metric": { 05:50:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG [...] 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG 05:50:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:51 DEBUG }, 05:50:51 DEBUG "value": [ 05:50:51 DEBUG 1713761451, 05:50:51 DEBUG "0" 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG ] 05:50:51 DEBUG } 05:50:51 DEBUG } 05:51:09 INFO 05:51:09 INFO [loop_until]: kubectl --namespace=xlou top pods 05:51:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:51:09 INFO [loop_until]: OK (rc = 0) 05:51:09 DEBUG --- stdout --- 05:51:09 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8790Mi am-767f6b67b5-qcqz2 7m 8837Mi am-767f6b67b5-vbltz 7m 8164Mi ds-cts-0 15m 5291Mi ds-cts-1 12m 11385Mi ds-cts-2 12m 10624Mi ds-idrepo-0 14m 4009Mi ds-idrepo-1 14m 3995Mi ds-idrepo-2 12m 3985Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1243Mi lodemon-7f68c498d9-gbc5j 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1015m 10473Mi 05:51:09 DEBUG --- stderr --- 05:51:09 DEBUG 05:51:17 INFO 05:51:17 INFO [loop_until]: kubectl --namespace=xlou top node 05:51:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:51:17 INFO [loop_until]: OK (rc = 0) 05:51:17 DEBUG --- stdout --- 05:51:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6234Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 8112Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 11518Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 3361Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 72m 0% 8533Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5194Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 11715Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3139Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 10384Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 7483Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 78m 0% 16545Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 88m 0% 8053Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 65m 0% 13341Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1081m 6% 20684Mi 35% 05:51:17 DEBUG --- stderr --- 05:51:17 DEBUG 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761508" 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761508" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761508" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761508" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761508" 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "result": [ 05:51:48 DEBUG { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "pod": "ds-cts-0" 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG "pod": "ds-cts-2" 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 1713761508, 05:51:48 DEBUG "0" 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 ERROR [http_cmd]: ERROR 05:51:48 ERROR --- status code --- 05:51:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:51:48 ERROR --- http response --- 05:51:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:51:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "result": [ 05:51:48 DEBUG { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "pod": "ds-idrepo-0" 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG "pod": "ds-idrepo-2" 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 1713761508, 05:51:48 DEBUG "0" 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "result": [ 05:51:48 DEBUG { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG "pod": "ds-cts-2" 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 1713761508, 05:51:48 DEBUG "0" 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761509" 05:51:49 INFO [http_cmd]: http status code OK 05:51:49 DEBUG --- status code --- 05:51:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:49 DEBUG --- http response --- 05:51:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:49 DEBUG ----- output ----- 05:51:49 DEBUG { 05:51:49 DEBUG "status": "success", 05:51:49 DEBUG "data": { 05:51:49 DEBUG "resultType": "vector", 05:51:49 DEBUG "result": [ 05:51:49 DEBUG { 05:51:49 DEBUG "metric": { 05:51:49 DEBUG "instance": "10.142.0.107:9100" 05:51:49 DEBUG }, 05:51:49 DEBUG "value": [ 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG [...] 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG "instance": "10.142.0.43:9100" 05:51:49 DEBUG }, 05:51:49 DEBUG "value": [ 05:51:49 DEBUG 1713761509, 05:51:49 DEBUG "5832697340928" 05:51:49 DEBUG ] 05:51:49 DEBUG } 05:51:49 DEBUG ] 05:51:49 DEBUG } 05:51:49 DEBUG } 05:51:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761509" 05:51:49 INFO [http_cmd]: http status code OK 05:51:49 DEBUG --- status code --- 05:51:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:49 DEBUG --- http response --- 05:51:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:49 DEBUG ----- output ----- 05:51:49 DEBUG { 05:51:49 DEBUG "status": "success", 05:51:49 DEBUG "data": { 05:51:49 DEBUG "resultType": "vector", 05:51:49 DEBUG "result": [ 05:51:49 DEBUG { 05:51:49 DEBUG "metric": { 05:51:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:49 DEBUG }, 05:51:49 DEBUG "value": [ 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG [...] 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:49 DEBUG }, 05:51:49 DEBUG "value": [ 05:51:49 DEBUG 1713761509, 05:51:49 DEBUG "0" 05:51:49 DEBUG ] 05:51:49 DEBUG } 05:51:49 DEBUG ] 05:51:49 DEBUG } 05:51:49 DEBUG } 05:51:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761509" 05:51:49 INFO [http_cmd]: http status code OK 05:51:49 DEBUG --- status code --- 05:51:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:49 DEBUG --- http response --- 05:51:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:49 DEBUG ----- output ----- 05:51:49 DEBUG { 05:51:49 DEBUG "status": "success", 05:51:49 DEBUG "data": { 05:51:49 DEBUG "resultType": "vector", 05:51:49 DEBUG "result": [ 05:51:49 DEBUG { 05:51:49 DEBUG "metric": { 05:51:49 DEBUG "pod": "ds-cts-0" 05:51:49 DEBUG }, 05:51:49 DEBUG "value": [ 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG [...] 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG 05:51:49 DEBUG "pod": "ds-cts-2" 05:51:49 DEBUG }, 05:51:49 DEBUG "value": [ 05:51:49 DEBUG 1713761509, 05:51:49 DEBUG "0" 05:51:49 DEBUG ] 05:51:49 DEBUG } 05:51:49 DEBUG ] 05:51:49 DEBUG } 05:51:49 DEBUG } 05:51:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761510" 05:51:50 INFO [http_cmd]: http status code OK 05:51:50 DEBUG --- status code --- 05:51:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:50 DEBUG --- http response --- 05:51:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:51:50 DEBUG ----- output ----- 05:51:50 DEBUG { 05:51:50 DEBUG "status": "success", 05:51:50 DEBUG "data": { 05:51:50 DEBUG "resultType": "vector", 05:51:50 DEBUG "result": [ 05:51:50 DEBUG { 05:51:50 DEBUG "metric": { 05:51:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:51:50 DEBUG }, 05:51:50 DEBUG "value": [ 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG [...] 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG "pod": "ds-cts-0" 05:51:50 DEBUG }, 05:51:50 DEBUG "value": [ 05:51:50 DEBUG 1713761510, 05:51:50 DEBUG "801.1997131120819" 05:51:50 DEBUG ] 05:51:50 DEBUG } 05:51:50 DEBUG ] 05:51:50 DEBUG } 05:51:50 DEBUG } 05:51:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761510" 05:51:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761510" 05:51:50 INFO [http_cmd]: http status code OK 05:51:50 DEBUG --- status code --- 05:51:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:50 DEBUG --- http response --- 05:51:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:50 DEBUG ----- output ----- 05:51:50 DEBUG { 05:51:50 DEBUG "status": "success", 05:51:50 DEBUG "data": { 05:51:50 DEBUG "resultType": "vector", 05:51:50 DEBUG "result": [ 05:51:50 DEBUG { 05:51:50 DEBUG "metric": { 05:51:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:50 DEBUG }, 05:51:50 DEBUG "value": [ 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG [...] 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:50 DEBUG }, 05:51:50 DEBUG "value": [ 05:51:50 DEBUG 1713761510, 05:51:50 DEBUG "991.915" 05:51:50 DEBUG ] 05:51:50 DEBUG } 05:51:50 DEBUG ] 05:51:50 DEBUG } 05:51:50 DEBUG } 05:51:50 INFO [http_cmd]: http status code OK 05:51:50 DEBUG --- status code --- 05:51:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:50 DEBUG --- http response --- 05:51:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:50 DEBUG ----- output ----- 05:51:50 DEBUG { 05:51:50 DEBUG "status": "success", 05:51:50 DEBUG "data": { 05:51:50 DEBUG "resultType": "vector", 05:51:50 DEBUG "result": [ 05:51:50 DEBUG { 05:51:50 DEBUG "metric": { 05:51:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:50 DEBUG }, 05:51:50 DEBUG "value": [ 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG [...] 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:50 DEBUG }, 05:51:50 DEBUG "value": [ 05:51:50 DEBUG 1713761510, 05:51:50 DEBUG "0" 05:51:50 DEBUG ] 05:51:50 DEBUG } 05:51:50 DEBUG ] 05:51:50 DEBUG } 05:51:50 DEBUG } 05:51:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761510" 05:51:50 INFO [http_cmd]: http status code OK 05:51:50 DEBUG --- status code --- 05:51:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:50 DEBUG --- http response --- 05:51:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:50 DEBUG ----- output ----- 05:51:50 DEBUG { 05:51:50 DEBUG "status": "success", 05:51:50 DEBUG "data": { 05:51:50 DEBUG "resultType": "vector", 05:51:50 DEBUG "result": [ 05:51:50 DEBUG { 05:51:50 DEBUG "metric": { 05:51:50 DEBUG "pod": "ds-idrepo-0" 05:51:50 DEBUG }, 05:51:50 DEBUG "value": [ 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG [...] 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG "pod": "ds-idrepo-2" 05:51:50 DEBUG }, 05:51:50 DEBUG "value": [ 05:51:50 DEBUG 1713761510, 05:51:50 DEBUG "0" 05:51:50 DEBUG ] 05:51:50 DEBUG } 05:51:50 DEBUG ] 05:51:50 DEBUG } 05:51:50 DEBUG } 05:51:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761510" 05:51:50 INFO [http_cmd]: http status code OK 05:51:50 DEBUG --- status code --- 05:51:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:50 DEBUG --- http response --- 05:51:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:50 DEBUG ----- output ----- 05:51:50 DEBUG { 05:51:50 DEBUG "status": "success", 05:51:50 DEBUG "data": { 05:51:50 DEBUG "resultType": "vector", 05:51:50 DEBUG "result": [ 05:51:50 DEBUG { 05:51:50 DEBUG "metric": { 05:51:50 DEBUG "instance": "10.142.0.107:9100" 05:51:50 DEBUG }, 05:51:50 DEBUG "value": [ 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG [...] 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG 05:51:50 DEBUG "instance": "10.142.0.43:9100" 05:51:50 DEBUG }, 05:51:50 DEBUG "value": [ 05:51:50 DEBUG 1713761510, 05:51:50 DEBUG "0.0005666666666930572" 05:51:50 DEBUG ] 05:51:50 DEBUG } 05:51:50 DEBUG ] 05:51:50 DEBUG } 05:51:50 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "instance": "10.142.0.107:9100" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "instance": "10.142.0.43:9100" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "0" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "instance": "10.142.0.107:9100" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "instance": "10.142.0.43:9100" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "38564.76666666666" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "11078692864" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "1.0289453794034449" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "pod": "ds-cts-0" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "0" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "0" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761511" 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "pod": "ds-cts-0" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "0.012698409072809466" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "0" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761511" 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "instance": "10.142.0.107:9100" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "instance": "10.142.0.43:9100" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "31303" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "pod": "ds-idrepo-0" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "pod": "ds-cts-2" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "0" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "pod": "ds-cts-0" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "pod": "ds-cts-2" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "0" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "0" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "pod": "ds-idrepo-0" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "pod": "ds-idrepo-2" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "0" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:51:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761511" 05:51:51 INFO [http_cmd]: http status code OK 05:51:51 DEBUG --- status code --- 05:51:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:51 DEBUG --- http response --- 05:51:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:51 DEBUG ----- output ----- 05:51:51 DEBUG { 05:51:51 DEBUG "status": "success", 05:51:51 DEBUG "data": { 05:51:51 DEBUG "resultType": "vector", 05:51:51 DEBUG "result": [ 05:51:51 DEBUG { 05:51:51 DEBUG "metric": { 05:51:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG [...] 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG 05:51:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:51 DEBUG }, 05:51:51 DEBUG "value": [ 05:51:51 DEBUG 1713761511, 05:51:51 DEBUG "0" 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG ] 05:51:51 DEBUG } 05:51:51 DEBUG } 05:52:09 INFO 05:52:09 INFO [loop_until]: kubectl --namespace=xlou top pods 05:52:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:52:09 INFO [loop_until]: OK (rc = 0) 05:52:09 DEBUG --- stdout --- 05:52:09 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8789Mi am-767f6b67b5-qcqz2 7m 8838Mi am-767f6b67b5-vbltz 7m 8164Mi ds-cts-0 14m 5291Mi ds-cts-1 13m 11386Mi ds-cts-2 15m 10624Mi ds-idrepo-0 14m 4010Mi ds-idrepo-1 13m 3995Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1243Mi lodemon-7f68c498d9-gbc5j 5m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 534m 10112Mi 05:52:09 DEBUG --- stderr --- 05:52:09 DEBUG 05:52:17 INFO 05:52:17 INFO [loop_until]: kubectl --namespace=xlou top node 05:52:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:52:17 INFO [loop_until]: OK (rc = 0) 05:52:17 DEBUG --- stdout --- 05:52:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6237Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 8112Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 61m 0% 11508Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3361Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 8528Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 5207Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 11717Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 74m 0% 3143Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 58m 0% 10390Mi 9% gke-xlou-cdm-ds-0f601751-j488 64m 0% 7486Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 74m 0% 16543Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 89m 0% 8033Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 13347Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 336m 2% 20327Mi 34% 05:52:17 DEBUG --- stderr --- 05:52:17 DEBUG 05:52:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761568" 05:52:48 INFO [http_cmd]: http status code OK 05:52:48 DEBUG --- status code --- 05:52:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:48 DEBUG --- http response --- 05:52:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761568" 05:52:48 INFO [http_cmd]: http status code OK 05:52:48 DEBUG --- status code --- 05:52:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:48 DEBUG --- http response --- 05:52:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:48 DEBUG ----- output ----- 05:52:48 DEBUG { 05:52:48 DEBUG "status": "success", 05:52:48 DEBUG "data": { 05:52:48 DEBUG "resultType": "vector", 05:52:48 DEBUG "result": [ 05:52:48 DEBUG { 05:52:48 DEBUG "metric": { 05:52:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:52:48 DEBUG }, 05:52:48 DEBUG "value": [ 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG [...] 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG "pod": "ds-cts-0" 05:52:48 DEBUG }, 05:52:48 DEBUG "value": [ 05:52:48 DEBUG 1713761568, 05:52:48 DEBUG "10205.980066445183" 05:52:48 DEBUG ] 05:52:48 DEBUG } 05:52:48 DEBUG ] 05:52:48 DEBUG } 05:52:48 DEBUG } 05:52:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761568" 05:52:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761568" 05:52:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761568" 05:52:48 ERROR [http_cmd]: ERROR 05:52:48 INFO [http_cmd]: http status code OK 05:52:48 ERROR --- status code --- 05:52:48 DEBUG --- status code --- 05:52:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:52:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:48 ERROR --- http response --- 05:52:48 DEBUG --- http response --- 05:52:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:52:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:52:48 DEBUG ----- output ----- 05:52:48 DEBUG { 05:52:48 DEBUG "status": "success", 05:52:48 DEBUG "data": { 05:52:48 DEBUG "resultType": "vector", 05:52:48 DEBUG "result": [ 05:52:48 DEBUG { 05:52:48 DEBUG "metric": { 05:52:48 DEBUG "pod": "ds-cts-0" 05:52:48 DEBUG }, 05:52:48 DEBUG "value": [ 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG [...] 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG "pod": "ds-cts-2" 05:52:48 DEBUG }, 05:52:48 DEBUG "value": [ 05:52:48 DEBUG 1713761568, 05:52:48 DEBUG "0" 05:52:48 DEBUG ] 05:52:48 DEBUG } 05:52:48 DEBUG ] 05:52:48 DEBUG } 05:52:48 DEBUG } 05:52:48 INFO [http_cmd]: http status code OK 05:52:48 DEBUG --- status code --- 05:52:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:48 DEBUG --- http response --- 05:52:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:48 DEBUG ----- output ----- 05:52:48 DEBUG { 05:52:48 DEBUG "status": "success", 05:52:48 DEBUG "data": { 05:52:48 DEBUG "resultType": "vector", 05:52:48 DEBUG "result": [ 05:52:48 DEBUG { 05:52:48 DEBUG "metric": { 05:52:48 DEBUG "pod": "ds-idrepo-0" 05:52:48 DEBUG }, 05:52:48 DEBUG "value": [ 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG [...] 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG 05:52:48 DEBUG "pod": "ds-idrepo-2" 05:52:48 DEBUG }, 05:52:48 DEBUG "value": [ 05:52:48 DEBUG 1713761568, 05:52:48 DEBUG "0" 05:52:48 DEBUG ] 05:52:48 DEBUG } 05:52:48 DEBUG ] 05:52:48 DEBUG } 05:52:48 DEBUG } 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761569" 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "result": [ 05:52:49 DEBUG { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "instance": "10.142.0.107:9100" 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG "instance": "10.142.0.43:9100" 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 1713761569, 05:52:49 DEBUG "5832698102784" 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761569" 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "result": [ 05:52:49 DEBUG { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 1713761569, 05:52:49 DEBUG "0" 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761569" 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "result": [ 05:52:49 DEBUG { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "pod": "ds-cts-0" 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG "pod": "ds-cts-2" 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 1713761569, 05:52:49 DEBUG "0" 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761570" 05:52:50 INFO [http_cmd]: http status code OK 05:52:50 DEBUG --- status code --- 05:52:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:50 DEBUG --- http response --- 05:52:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:52:50 DEBUG ----- output ----- 05:52:50 DEBUG { 05:52:50 DEBUG "status": "success", 05:52:50 DEBUG "data": { 05:52:50 DEBUG "resultType": "vector", 05:52:50 DEBUG "result": [ 05:52:50 DEBUG { 05:52:50 DEBUG "metric": { 05:52:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:52:50 DEBUG }, 05:52:50 DEBUG "value": [ 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG [...] 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:52:50 DEBUG }, 05:52:50 DEBUG "value": [ 05:52:50 DEBUG 1713761570, 05:52:50 DEBUG "2292.6379560121163" 05:52:50 DEBUG ] 05:52:50 DEBUG } 05:52:50 DEBUG ] 05:52:50 DEBUG } 05:52:50 DEBUG } 05:52:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761570" 05:52:50 INFO [http_cmd]: http status code OK 05:52:50 DEBUG --- status code --- 05:52:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:50 DEBUG --- http response --- 05:52:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:50 DEBUG ----- output ----- 05:52:50 DEBUG { 05:52:50 DEBUG "status": "success", 05:52:50 DEBUG "data": { 05:52:50 DEBUG "resultType": "vector", 05:52:50 DEBUG "result": [ 05:52:50 DEBUG { 05:52:50 DEBUG "metric": { 05:52:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:50 DEBUG }, 05:52:50 DEBUG "value": [ 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG [...] 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:50 DEBUG }, 05:52:50 DEBUG "value": [ 05:52:50 DEBUG 1713761570, 05:52:50 DEBUG "0" 05:52:50 DEBUG ] 05:52:50 DEBUG } 05:52:50 DEBUG ] 05:52:50 DEBUG } 05:52:50 DEBUG } 05:52:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761570" 05:52:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761570" 05:52:50 INFO [http_cmd]: http status code OK 05:52:50 INFO [http_cmd]: http status code OK 05:52:50 DEBUG --- status code --- 05:52:50 DEBUG --- status code --- 05:52:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:50 DEBUG --- http response --- 05:52:50 DEBUG --- http response --- 05:52:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:50 DEBUG ----- output ----- 05:52:50 DEBUG { 05:52:50 DEBUG "status": "success", 05:52:50 DEBUG "data": { 05:52:50 DEBUG "resultType": "vector", 05:52:50 DEBUG "result": [ 05:52:50 DEBUG { 05:52:50 DEBUG "metric": { 05:52:50 DEBUG "pod": "ds-idrepo-0" 05:52:50 DEBUG }, 05:52:50 DEBUG "value": [ 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG [...] 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG "pod": "ds-idrepo-2" 05:52:50 DEBUG }, 05:52:50 DEBUG "value": [ 05:52:50 DEBUG 1713761570, 05:52:50 DEBUG "0" 05:52:50 DEBUG ] 05:52:50 DEBUG } 05:52:50 DEBUG ] 05:52:50 DEBUG } 05:52:50 DEBUG } 05:52:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:50 DEBUG ----- output ----- 05:52:50 DEBUG { 05:52:50 DEBUG "status": "success", 05:52:50 DEBUG "data": { 05:52:50 DEBUG "resultType": "vector", 05:52:50 DEBUG "result": [ 05:52:50 DEBUG { 05:52:50 DEBUG "metric": { 05:52:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:50 DEBUG }, 05:52:50 DEBUG "value": [ 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG [...] 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:50 DEBUG }, 05:52:50 DEBUG "value": [ 05:52:50 DEBUG 1713761570, 05:52:50 DEBUG "991.9155000000001" 05:52:50 DEBUG ] 05:52:50 DEBUG } 05:52:50 DEBUG ] 05:52:50 DEBUG } 05:52:50 DEBUG } 05:52:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761570" 05:52:50 INFO [http_cmd]: http status code OK 05:52:50 DEBUG --- status code --- 05:52:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:50 DEBUG --- http response --- 05:52:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:50 DEBUG ----- output ----- 05:52:50 DEBUG { 05:52:50 DEBUG "status": "success", 05:52:50 DEBUG "data": { 05:52:50 DEBUG "resultType": "vector", 05:52:50 DEBUG "result": [ 05:52:50 DEBUG { 05:52:50 DEBUG "metric": { 05:52:50 DEBUG "instance": "10.142.0.107:9100" 05:52:50 DEBUG }, 05:52:50 DEBUG "value": [ 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG [...] 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG 05:52:50 DEBUG "instance": "10.142.0.43:9100" 05:52:50 DEBUG }, 05:52:50 DEBUG "value": [ 05:52:50 DEBUG 1713761570, 05:52:50 DEBUG "0.000533333333360512" 05:52:50 DEBUG ] 05:52:50 DEBUG } 05:52:50 DEBUG ] 05:52:50 DEBUG } 05:52:50 DEBUG } 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761571" 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761571" 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "instance": "10.142.0.107:9100" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "instance": "10.142.0.43:9100" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "38149.86666666667" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "instance": "10.142.0.107:9100" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "instance": "10.142.0.43:9100" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "0.0003333333333330302" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761571" 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "11147100160" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761571" 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "0.9655628762908612" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761571" 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761571" 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761571" 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761571" 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "106046642.96062163" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "0" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:52:51 DEBUG }, 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:51 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:52:51 DEBUG }, 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "0" 05:52:51 DEBUG "1.1363197218822572" 05:52:51 DEBUG ] 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761571" 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761571" 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761571" 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "pod": "ds-idrepo-0" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "pod": "ds-cts-2" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "0" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "instance": "10.142.0.107:9100" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "instance": "10.142.0.43:9100" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "31094.666666666664" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761571" 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "pod": "ds-idrepo-0" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "pod": "ds-idrepo-2" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "0" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "pod": "ds-cts-0" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "pod": "ds-cts-2" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "0" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761571" 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "0" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:52:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761571" 05:52:51 INFO [http_cmd]: http status code OK 05:52:51 DEBUG --- status code --- 05:52:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:51 DEBUG --- http response --- 05:52:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:51 DEBUG ----- output ----- 05:52:51 DEBUG { 05:52:51 DEBUG "status": "success", 05:52:51 DEBUG "data": { 05:52:51 DEBUG "resultType": "vector", 05:52:51 DEBUG "result": [ 05:52:51 DEBUG { 05:52:51 DEBUG "metric": { 05:52:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG [...] 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG 05:52:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:51 DEBUG }, 05:52:51 DEBUG "value": [ 05:52:51 DEBUG 1713761571, 05:52:51 DEBUG "0" 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG ] 05:52:51 DEBUG } 05:52:51 DEBUG } 05:53:09 INFO 05:53:09 INFO [loop_until]: kubectl --namespace=xlou top pods 05:53:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:53:09 INFO [loop_until]: OK (rc = 0) 05:53:09 DEBUG --- stdout --- 05:53:09 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8790Mi am-767f6b67b5-qcqz2 7m 8838Mi am-767f6b67b5-vbltz 7m 8164Mi ds-cts-0 13m 5290Mi ds-cts-1 14m 11385Mi ds-cts-2 12m 10624Mi ds-idrepo-0 14m 4010Mi ds-idrepo-1 13m 3994Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 2m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1061m 10606Mi 05:53:09 DEBUG --- stderr --- 05:53:09 DEBUG 05:53:17 INFO 05:53:17 INFO [loop_until]: kubectl --namespace=xlou top node 05:53:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:53:17 INFO [loop_until]: OK (rc = 0) 05:53:17 DEBUG --- stdout --- 05:53:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6229Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 8108Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 11498Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3347Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 72m 0% 8523Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 5203Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 11715Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 74m 0% 3134Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 59m 0% 10387Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 7479Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 77m 0% 16546Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 96m 0% 8037Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 68m 0% 13344Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1127m 7% 20868Mi 35% 05:53:17 DEBUG --- stderr --- 05:53:17 DEBUG 05:53:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761628" 05:53:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761628" 05:53:48 INFO [http_cmd]: http status code OK 05:53:48 DEBUG --- status code --- 05:53:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:48 DEBUG --- http response --- 05:53:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:48 ERROR [http_cmd]: ERROR 05:53:48 ERROR --- status code --- 05:53:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:53:48 ERROR --- http response --- 05:53:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:53:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:53:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761628" 05:53:48 INFO [http_cmd]: http status code OK 05:53:48 DEBUG --- status code --- 05:53:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:48 DEBUG --- http response --- 05:53:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:53:48 DEBUG ----- output ----- 05:53:48 DEBUG { 05:53:48 DEBUG "status": "success", 05:53:48 DEBUG "data": { 05:53:48 DEBUG "resultType": "vector", 05:53:48 DEBUG "result": [ 05:53:48 DEBUG { 05:53:48 DEBUG "metric": { 05:53:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:53:48 DEBUG }, 05:53:48 DEBUG "value": [ 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG [...] 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG "pod": "ds-idrepo-0" 05:53:48 DEBUG }, 05:53:48 DEBUG "value": [ 05:53:48 DEBUG 1713761628, 05:53:48 DEBUG "0" 05:53:48 DEBUG ] 05:53:48 DEBUG } 05:53:48 DEBUG ] 05:53:48 DEBUG } 05:53:48 DEBUG } 05:53:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761628" 05:53:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761628" 05:53:48 INFO [http_cmd]: http status code OK 05:53:48 INFO [http_cmd]: http status code OK 05:53:48 DEBUG --- status code --- 05:53:48 DEBUG --- status code --- 05:53:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:48 DEBUG --- http response --- 05:53:48 DEBUG --- http response --- 05:53:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:48 DEBUG ----- output ----- 05:53:48 DEBUG { 05:53:48 DEBUG "status": "success", 05:53:48 DEBUG "data": { 05:53:48 DEBUG "resultType": "vector", 05:53:48 DEBUG "result": [ 05:53:48 DEBUG { 05:53:48 DEBUG "metric": { 05:53:48 DEBUG "pod": "ds-idrepo-0" 05:53:48 DEBUG }, 05:53:48 DEBUG "value": [ 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG [...] 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG "pod": "ds-idrepo-2" 05:53:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:48 DEBUG }, 05:53:48 DEBUG ----- output ----- 05:53:48 DEBUG "value": [ 05:53:48 DEBUG { 05:53:48 DEBUG 1713761628, 05:53:48 DEBUG "status": "success", 05:53:48 DEBUG "0" 05:53:48 DEBUG "data": { 05:53:48 DEBUG ] 05:53:48 DEBUG "resultType": "vector", 05:53:48 DEBUG } 05:53:48 DEBUG "result": [ 05:53:48 DEBUG ] 05:53:48 DEBUG { 05:53:48 DEBUG } 05:53:48 DEBUG "metric": { 05:53:48 DEBUG } 05:53:48 DEBUG "pod": "ds-cts-0" 05:53:48 DEBUG }, 05:53:48 DEBUG "value": [ 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG [...] 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG 05:53:48 DEBUG "pod": "ds-cts-2" 05:53:48 DEBUG }, 05:53:48 DEBUG "value": [ 05:53:48 DEBUG 1713761628, 05:53:48 DEBUG "0" 05:53:48 DEBUG ] 05:53:48 DEBUG } 05:53:48 DEBUG ] 05:53:48 DEBUG } 05:53:48 DEBUG } 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761629" 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG { 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "data": { 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "result": [ 05:53:49 DEBUG { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "instance": "10.142.0.107:9100" 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "instance": "10.142.0.43:9100" 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 1713761629, 05:53:49 DEBUG "5832698860544" 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761629" 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG { 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "data": { 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "result": [ 05:53:49 DEBUG { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 1713761629, 05:53:49 DEBUG "0" 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761629" 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG { 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "data": { 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "result": [ 05:53:49 DEBUG { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "pod": "ds-cts-0" 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "pod": "ds-cts-2" 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 1713761629, 05:53:49 DEBUG "0" 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761630" 05:53:50 INFO [http_cmd]: http status code OK 05:53:50 DEBUG --- status code --- 05:53:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:50 DEBUG --- http response --- 05:53:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:50 DEBUG ----- output ----- 05:53:50 DEBUG { 05:53:50 DEBUG "status": "success", 05:53:50 DEBUG "data": { 05:53:50 DEBUG "resultType": "vector", 05:53:50 DEBUG "result": [ 05:53:50 DEBUG { 05:53:50 DEBUG "metric": { 05:53:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:53:50 DEBUG }, 05:53:50 DEBUG "value": [ 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG [...] 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:53:50 DEBUG }, 05:53:50 DEBUG "value": [ 05:53:50 DEBUG 1713761630, 05:53:50 DEBUG "0" 05:53:50 DEBUG ] 05:53:50 DEBUG } 05:53:50 DEBUG ] 05:53:50 DEBUG } 05:53:50 DEBUG } 05:53:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761630" 05:53:50 INFO [http_cmd]: http status code OK 05:53:50 DEBUG --- status code --- 05:53:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:50 DEBUG --- http response --- 05:53:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:50 DEBUG ----- output ----- 05:53:50 DEBUG { 05:53:50 DEBUG "status": "success", 05:53:50 DEBUG "data": { 05:53:50 DEBUG "resultType": "vector", 05:53:50 DEBUG "result": [ 05:53:50 DEBUG { 05:53:50 DEBUG "metric": { 05:53:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:50 DEBUG }, 05:53:50 DEBUG "value": [ 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG [...] 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:50 DEBUG }, 05:53:50 DEBUG "value": [ 05:53:50 DEBUG 1713761630, 05:53:50 DEBUG "0" 05:53:50 DEBUG ] 05:53:50 DEBUG } 05:53:50 DEBUG ] 05:53:50 DEBUG } 05:53:50 DEBUG } 05:53:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761630" 05:53:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761630" 05:53:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761630" 05:53:50 INFO [http_cmd]: http status code OK 05:53:50 INFO [http_cmd]: http status code OK 05:53:50 DEBUG --- status code --- 05:53:50 INFO [http_cmd]: http status code OK 05:53:50 DEBUG --- status code --- 05:53:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:50 DEBUG --- status code --- 05:53:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:50 DEBUG --- http response --- 05:53:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:50 DEBUG --- http response --- 05:53:50 DEBUG --- http response --- 05:53:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:50 DEBUG ----- output ----- 05:53:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:50 DEBUG ----- output ----- 05:53:50 DEBUG { 05:53:50 DEBUG ----- output ----- 05:53:50 DEBUG { 05:53:50 DEBUG "status": "success", 05:53:50 DEBUG { 05:53:50 DEBUG "status": "success", 05:53:50 DEBUG "data": { 05:53:50 DEBUG "status": "success", 05:53:50 DEBUG "data": { 05:53:50 DEBUG "resultType": "vector", 05:53:50 DEBUG "data": { 05:53:50 DEBUG "resultType": "vector", 05:53:50 DEBUG "result": [ 05:53:50 DEBUG "resultType": "vector", 05:53:50 DEBUG "result": [ 05:53:50 DEBUG { 05:53:50 DEBUG "result": [ 05:53:50 DEBUG { 05:53:50 DEBUG "metric": { 05:53:50 DEBUG { 05:53:50 DEBUG "metric": { 05:53:50 DEBUG "pod": "ds-idrepo-0" 05:53:50 DEBUG "metric": { 05:53:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:50 DEBUG }, 05:53:50 DEBUG "instance": "10.142.0.107:9100" 05:53:50 DEBUG }, 05:53:50 DEBUG "value": [ 05:53:50 DEBUG }, 05:53:50 DEBUG "value": [ 05:53:50 DEBUG 05:53:50 DEBUG "value": [ 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG [...] 05:53:50 DEBUG 05:53:50 DEBUG [...] 05:53:50 DEBUG 05:53:50 DEBUG [...] 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG 05:53:50 DEBUG "pod": "ds-idrepo-2" 05:53:50 DEBUG 05:53:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:50 DEBUG }, 05:53:50 DEBUG "instance": "10.142.0.43:9100" 05:53:50 DEBUG }, 05:53:50 DEBUG "value": [ 05:53:50 DEBUG }, 05:53:50 DEBUG "value": [ 05:53:50 DEBUG 1713761630, 05:53:50 DEBUG "value": [ 05:53:50 DEBUG 1713761630, 05:53:50 DEBUG "0" 05:53:50 DEBUG 1713761630, 05:53:50 DEBUG "991.916" 05:53:50 DEBUG ] 05:53:50 DEBUG "0.0005666666666930572" 05:53:50 DEBUG ] 05:53:50 DEBUG } 05:53:50 DEBUG ] 05:53:50 DEBUG } 05:53:50 DEBUG ] 05:53:50 DEBUG } 05:53:50 DEBUG ] 05:53:50 DEBUG } 05:53:50 DEBUG ] 05:53:50 DEBUG } 05:53:50 DEBUG } 05:53:50 DEBUG } 05:53:50 DEBUG } 05:53:50 DEBUG } 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761631" 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "instance": "10.142.0.107:9100" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "instance": "10.142.0.43:9100" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "37593.03333333333" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761631" 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "instance": "10.142.0.107:9100" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "instance": "10.142.0.43:9100" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "0" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761631" 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "11224821760" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761631" 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "1.0267300178503422" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761631" 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "0" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761631" 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "0" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761631" 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761631" 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "0" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "0.007005595262472694" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761631" 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761631" 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761631" 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761631" 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "pod": "ds-idrepo-0" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "pod": "ds-cts-2" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "0" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "pod": "ds-cts-0" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "pod": "ds-cts-2" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "0" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "instance": "10.142.0.107:9100" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "instance": "10.142.0.43:9100" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "31001.73333333333" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "pod": "ds-idrepo-0" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "pod": "ds-idrepo-2" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "0" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761631" 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "0" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:53:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761631" 05:53:51 INFO [http_cmd]: http status code OK 05:53:51 DEBUG --- status code --- 05:53:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:51 DEBUG --- http response --- 05:53:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:51 DEBUG ----- output ----- 05:53:51 DEBUG { 05:53:51 DEBUG "status": "success", 05:53:51 DEBUG "data": { 05:53:51 DEBUG "resultType": "vector", 05:53:51 DEBUG "result": [ 05:53:51 DEBUG { 05:53:51 DEBUG "metric": { 05:53:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG [...] 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG 05:53:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:51 DEBUG }, 05:53:51 DEBUG "value": [ 05:53:51 DEBUG 1713761631, 05:53:51 DEBUG "0" 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG ] 05:53:51 DEBUG } 05:53:51 DEBUG } 05:54:09 INFO 05:54:09 INFO [loop_until]: kubectl --namespace=xlou top pods 05:54:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:54:09 INFO [loop_until]: OK (rc = 0) 05:54:09 DEBUG --- stdout --- 05:54:09 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8789Mi am-767f6b67b5-qcqz2 9m 8837Mi am-767f6b67b5-vbltz 7m 8164Mi ds-cts-0 14m 5290Mi ds-cts-1 13m 11387Mi ds-cts-2 13m 10624Mi ds-idrepo-0 13m 4010Mi ds-idrepo-1 11m 3995Mi ds-idrepo-2 13m 3985Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 1m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-68c5d5b884-p97w5 1040m 10615Mi 05:54:09 DEBUG --- stderr --- 05:54:09 DEBUG 05:54:17 INFO 05:54:17 INFO [loop_until]: kubectl --namespace=xlou top node 05:54:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:54:17 INFO [loop_until]: OK (rc = 0) 05:54:17 DEBUG --- stdout --- 05:54:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6227Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 8107Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 11502Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3342Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 8532Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 11718Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 74m 0% 3137Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 59m 0% 10374Mi 9% gke-xlou-cdm-ds-0f601751-j488 69m 0% 7478Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 80m 0% 16554Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 8040Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 13347Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1069m 6% 20876Mi 35% 05:54:17 DEBUG --- stderr --- 05:54:17 DEBUG 05:54:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761688" 05:54:48 ERROR [http_cmd]: ERROR 05:54:48 ERROR --- status code --- 05:54:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:54:48 ERROR --- http response --- 05:54:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:54:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:54:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761688" 05:54:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761688" 05:54:48 INFO [http_cmd]: http status code OK 05:54:48 DEBUG --- status code --- 05:54:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:48 DEBUG --- http response --- 05:54:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:48 INFO [http_cmd]: http status code OK 05:54:48 DEBUG --- status code --- 05:54:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:48 DEBUG --- http response --- 05:54:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:54:48 DEBUG ----- output ----- 05:54:48 DEBUG { 05:54:48 DEBUG "status": "success", 05:54:48 DEBUG "data": { 05:54:48 DEBUG "resultType": "vector", 05:54:48 DEBUG "result": [ 05:54:48 DEBUG { 05:54:48 DEBUG "metric": { 05:54:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:54:48 DEBUG }, 05:54:48 DEBUG "value": [ 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG [...] 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG "pod": "ds-cts-2" 05:54:48 DEBUG }, 05:54:48 DEBUG "value": [ 05:54:48 DEBUG 1713761688, 05:54:48 DEBUG "297017558.32873553" 05:54:48 DEBUG ] 05:54:48 DEBUG } 05:54:48 DEBUG ] 05:54:48 DEBUG } 05:54:48 DEBUG } 05:54:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761688" 05:54:48 INFO [http_cmd]: http status code OK 05:54:48 DEBUG --- status code --- 05:54:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:48 DEBUG --- http response --- 05:54:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:48 DEBUG ----- output ----- 05:54:48 DEBUG { 05:54:48 DEBUG "status": "success", 05:54:48 DEBUG "data": { 05:54:48 DEBUG "resultType": "vector", 05:54:48 DEBUG "result": [ 05:54:48 DEBUG { 05:54:48 DEBUG "metric": { 05:54:48 DEBUG "pod": "ds-idrepo-0" 05:54:48 DEBUG }, 05:54:48 DEBUG "value": [ 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG [...] 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG "pod": "ds-idrepo-2" 05:54:48 DEBUG }, 05:54:48 DEBUG "value": [ 05:54:48 DEBUG 1713761688, 05:54:48 DEBUG "0" 05:54:48 DEBUG ] 05:54:48 DEBUG } 05:54:48 DEBUG ] 05:54:48 DEBUG } 05:54:48 DEBUG } 05:54:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761688" 05:54:48 INFO [http_cmd]: http status code OK 05:54:48 DEBUG --- status code --- 05:54:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:48 DEBUG --- http response --- 05:54:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:48 DEBUG ----- output ----- 05:54:48 DEBUG { 05:54:48 DEBUG "status": "success", 05:54:48 DEBUG "data": { 05:54:48 DEBUG "resultType": "vector", 05:54:48 DEBUG "result": [ 05:54:48 DEBUG { 05:54:48 DEBUG "metric": { 05:54:48 DEBUG "pod": "ds-cts-0" 05:54:48 DEBUG }, 05:54:48 DEBUG "value": [ 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG [...] 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG 05:54:48 DEBUG "pod": "ds-cts-2" 05:54:48 DEBUG }, 05:54:48 DEBUG "value": [ 05:54:48 DEBUG 1713761688, 05:54:48 DEBUG "0" 05:54:48 DEBUG ] 05:54:48 DEBUG } 05:54:48 DEBUG ] 05:54:48 DEBUG } 05:54:48 DEBUG } 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761689" 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG { 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "data": { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "result": [ 05:54:49 DEBUG { 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "instance": "10.142.0.107:9100" 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "instance": "10.142.0.43:9100" 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 1713761689, 05:54:49 DEBUG "5832699528192" 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761689" 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG { 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "data": { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "result": [ 05:54:49 DEBUG { 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 1713761689, 05:54:49 DEBUG "0" 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761689" 05:54:50 INFO [http_cmd]: http status code OK 05:54:50 DEBUG --- status code --- 05:54:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:50 DEBUG --- http response --- 05:54:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:50 DEBUG ----- output ----- 05:54:50 DEBUG { 05:54:50 DEBUG "status": "success", 05:54:50 DEBUG "data": { 05:54:50 DEBUG "resultType": "vector", 05:54:50 DEBUG "result": [ 05:54:50 DEBUG { 05:54:50 DEBUG "metric": { 05:54:50 DEBUG "pod": "ds-cts-0" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG [...] 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG "pod": "ds-cts-2" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 1713761689, 05:54:50 DEBUG "0" 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG } 05:54:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761690" 05:54:50 INFO [http_cmd]: http status code OK 05:54:50 DEBUG --- status code --- 05:54:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:50 DEBUG --- http response --- 05:54:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:54:50 DEBUG ----- output ----- 05:54:50 DEBUG { 05:54:50 DEBUG "status": "success", 05:54:50 DEBUG "data": { 05:54:50 DEBUG "resultType": "vector", 05:54:50 DEBUG "result": [ 05:54:50 DEBUG { 05:54:50 DEBUG "metric": { 05:54:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG [...] 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG "pod": "ds-idrepo-0" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 1713761690, 05:54:50 DEBUG "4478.806200553161" 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG } 05:54:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761690" 05:54:50 INFO [http_cmd]: http status code OK 05:54:50 DEBUG --- status code --- 05:54:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:50 DEBUG --- http response --- 05:54:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:50 DEBUG ----- output ----- 05:54:50 DEBUG { 05:54:50 DEBUG "status": "success", 05:54:50 DEBUG "data": { 05:54:50 DEBUG "resultType": "vector", 05:54:50 DEBUG "result": [ 05:54:50 DEBUG { 05:54:50 DEBUG "metric": { 05:54:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG [...] 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 1713761690, 05:54:50 DEBUG "0" 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG } 05:54:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761690" 05:54:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761690" 05:54:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761690" 05:54:50 INFO [http_cmd]: http status code OK 05:54:50 DEBUG --- status code --- 05:54:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:50 DEBUG --- http response --- 05:54:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:50 DEBUG ----- output ----- 05:54:50 DEBUG { 05:54:50 DEBUG "status": "success", 05:54:50 DEBUG "data": { 05:54:50 DEBUG "resultType": "vector", 05:54:50 DEBUG "result": [ 05:54:50 DEBUG { 05:54:50 DEBUG "metric": { 05:54:50 DEBUG "pod": "ds-idrepo-0" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG [...] 05:54:50 DEBUG 05:54:50 INFO [http_cmd]: http status code OK 05:54:50 DEBUG 05:54:50 DEBUG --- status code --- 05:54:50 DEBUG 05:54:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:50 DEBUG "pod": "ds-idrepo-2" 05:54:50 DEBUG --- http response --- 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 1713761690, 05:54:50 DEBUG "0" 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG } 05:54:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:50 DEBUG ----- output ----- 05:54:50 DEBUG { 05:54:50 DEBUG "status": "success", 05:54:50 DEBUG "data": { 05:54:50 DEBUG "resultType": "vector", 05:54:50 DEBUG "result": [ 05:54:50 DEBUG { 05:54:50 DEBUG "metric": { 05:54:50 DEBUG "instance": "10.142.0.107:9100" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG [...] 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG "instance": "10.142.0.43:9100" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 1713761690, 05:54:50 DEBUG "0.0005000000000260721" 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG } 05:54:50 INFO [http_cmd]: http status code OK 05:54:50 DEBUG --- status code --- 05:54:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:50 DEBUG --- http response --- 05:54:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:50 DEBUG ----- output ----- 05:54:50 DEBUG { 05:54:50 DEBUG "status": "success", 05:54:50 DEBUG "data": { 05:54:50 DEBUG "resultType": "vector", 05:54:50 DEBUG "result": [ 05:54:50 DEBUG { 05:54:50 DEBUG "metric": { 05:54:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG [...] 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG 05:54:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:50 DEBUG }, 05:54:50 DEBUG "value": [ 05:54:50 DEBUG 1713761690, 05:54:50 DEBUG "991.916" 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG ] 05:54:50 DEBUG } 05:54:50 DEBUG } 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761691" 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "instance": "10.142.0.107:9100" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "instance": "10.142.0.43:9100" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "0" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761691" 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "instance": "10.142.0.107:9100" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "instance": "10.142.0.43:9100" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "38165.399999999994" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761691" 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "11228393472" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761691" 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "1.0305506827404294" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761691" 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "pod": "ds-idrepo-0" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "0" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761691" 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "0" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761691" 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761691" 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761691" 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG { 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "0" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "pod": "ds-idrepo-0" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "0.012939120085736602" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "pod": "ds-idrepo-0" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "pod": "ds-idrepo-2" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "0" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761691" 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761691" 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761691" 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "pod": "ds-idrepo-0" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "pod": "ds-cts-2" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "0" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "pod": "ds-cts-0" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "pod": "ds-cts-2" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "0" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "instance": "10.142.0.107:9100" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "instance": "10.142.0.43:9100" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "32695.966666666667" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761691" 05:54:51 INFO [http_cmd]: http status code OK 05:54:51 DEBUG --- status code --- 05:54:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:51 DEBUG --- http response --- 05:54:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:51 DEBUG ----- output ----- 05:54:51 DEBUG { 05:54:51 DEBUG "status": "success", 05:54:51 DEBUG "data": { 05:54:51 DEBUG "resultType": "vector", 05:54:51 DEBUG "result": [ 05:54:51 DEBUG { 05:54:51 DEBUG "metric": { 05:54:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG [...] 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG 05:54:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:51 DEBUG }, 05:54:51 DEBUG "value": [ 05:54:51 DEBUG 1713761691, 05:54:51 DEBUG "0" 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG ] 05:54:51 DEBUG } 05:54:51 DEBUG } 05:54:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761692" 05:54:52 INFO [http_cmd]: http status code OK 05:54:52 DEBUG --- status code --- 05:54:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:52 DEBUG --- http response --- 05:54:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:52 DEBUG ----- output ----- 05:54:52 DEBUG { 05:54:52 DEBUG "status": "success", 05:54:52 DEBUG "data": { 05:54:52 DEBUG "resultType": "vector", 05:54:52 DEBUG "result": [ 05:54:52 DEBUG { 05:54:52 DEBUG "metric": { 05:54:52 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:52 DEBUG }, 05:54:52 DEBUG "value": [ 05:54:52 DEBUG 05:54:52 DEBUG 05:54:52 DEBUG 05:54:52 DEBUG [...] 05:54:52 DEBUG 05:54:52 DEBUG 05:54:52 DEBUG 05:54:52 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:52 DEBUG }, 05:54:52 DEBUG "value": [ 05:54:52 DEBUG 1713761692, 05:54:52 DEBUG "0" 05:54:52 DEBUG ] 05:54:52 DEBUG } 05:54:52 DEBUG ] 05:54:52 DEBUG } 05:54:52 DEBUG } 05:55:09 INFO 05:55:09 INFO [loop_until]: kubectl --namespace=xlou top pods 05:55:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:55:09 INFO [loop_until]: OK (rc = 0) 05:55:09 DEBUG --- stdout --- 05:55:09 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8790Mi am-767f6b67b5-qcqz2 7m 8838Mi am-767f6b67b5-vbltz 7m 8165Mi ds-cts-0 15m 5290Mi ds-cts-1 14m 11385Mi ds-cts-2 13m 10624Mi ds-idrepo-0 14m 4009Mi ds-idrepo-1 16m 3994Mi ds-idrepo-2 12m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-68c5d5b884-p97w5 1031m 10615Mi 05:55:09 DEBUG --- stderr --- 05:55:09 DEBUG 05:55:17 INFO 05:55:17 INFO [loop_until]: kubectl --namespace=xlou top node 05:55:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:55:17 INFO [loop_until]: OK (rc = 0) 05:55:17 DEBUG --- stdout --- 05:55:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6219Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 75m 0% 8095Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 11503Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3364Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 73m 0% 8529Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 5232Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 11710Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 3136Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 59m 0% 10380Mi 9% gke-xlou-cdm-ds-0f601751-j488 69m 0% 7492Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 80m 0% 16552Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 95m 0% 8045Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 70m 0% 13345Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1085m 6% 20875Mi 35% 05:55:17 DEBUG --- stderr --- 05:55:17 DEBUG 05:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713761748" 05:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761748" 05:55:48 ERROR [http_cmd]: ERROR 05:55:48 ERROR --- status code --- 05:55:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:55:48 ERROR --- http response --- 05:55:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:55:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:55:48 INFO [http_cmd]: http status code OK 05:55:48 DEBUG --- status code --- 05:55:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:48 DEBUG --- http response --- 05:55:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761748" 05:55:48 INFO [http_cmd]: http status code OK 05:55:48 DEBUG --- status code --- 05:55:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:48 DEBUG --- http response --- 05:55:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:55:48 DEBUG ----- output ----- 05:55:48 DEBUG { 05:55:48 DEBUG "status": "success", 05:55:48 DEBUG "data": { 05:55:48 DEBUG "resultType": "vector", 05:55:48 DEBUG "result": [ 05:55:48 DEBUG { 05:55:48 DEBUG "metric": { 05:55:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:55:48 DEBUG }, 05:55:48 DEBUG "value": [ 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG [...] 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:55:48 DEBUG }, 05:55:48 DEBUG "value": [ 05:55:48 DEBUG 1713761748, 05:55:48 DEBUG "6593.453764979431" 05:55:48 DEBUG ] 05:55:48 DEBUG } 05:55:48 DEBUG ] 05:55:48 DEBUG } 05:55:48 DEBUG } 05:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761748" 05:55:48 INFO [http_cmd]: http status code OK 05:55:48 DEBUG --- status code --- 05:55:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:48 DEBUG --- http response --- 05:55:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:48 DEBUG ----- output ----- 05:55:48 DEBUG { 05:55:48 DEBUG "status": "success", 05:55:48 DEBUG "data": { 05:55:48 DEBUG "resultType": "vector", 05:55:48 DEBUG "result": [ 05:55:48 DEBUG { 05:55:48 DEBUG "metric": { 05:55:48 DEBUG "pod": "ds-idrepo-0" 05:55:48 DEBUG }, 05:55:48 DEBUG "value": [ 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG [...] 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG "pod": "ds-idrepo-2" 05:55:48 DEBUG }, 05:55:48 DEBUG "value": [ 05:55:48 DEBUG 1713761748, 05:55:48 DEBUG "0" 05:55:48 DEBUG ] 05:55:48 DEBUG } 05:55:48 DEBUG ] 05:55:48 DEBUG } 05:55:48 DEBUG } 05:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761748" 05:55:48 INFO [http_cmd]: http status code OK 05:55:48 DEBUG --- status code --- 05:55:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:48 DEBUG --- http response --- 05:55:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:48 DEBUG ----- output ----- 05:55:48 DEBUG { 05:55:48 DEBUG "status": "success", 05:55:48 DEBUG "data": { 05:55:48 DEBUG "resultType": "vector", 05:55:48 DEBUG "result": [ 05:55:48 DEBUG { 05:55:48 DEBUG "metric": { 05:55:48 DEBUG "pod": "ds-cts-0" 05:55:48 DEBUG }, 05:55:48 DEBUG "value": [ 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG [...] 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG 05:55:48 DEBUG "pod": "ds-cts-2" 05:55:48 DEBUG }, 05:55:48 DEBUG "value": [ 05:55:48 DEBUG 1713761748, 05:55:48 DEBUG "0" 05:55:48 DEBUG ] 05:55:48 DEBUG } 05:55:48 DEBUG ] 05:55:48 DEBUG } 05:55:48 DEBUG } 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761749" 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "data": { 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "result": [ 05:55:49 DEBUG { 05:55:49 DEBUG "metric": { 05:55:49 DEBUG "instance": "10.142.0.107:9100" 05:55:49 DEBUG }, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "instance": "10.142.0.43:9100" 05:55:49 DEBUG }, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 1713761749, 05:55:49 DEBUG "5832700279808" 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761750" 05:55:50 INFO [http_cmd]: http status code OK 05:55:50 DEBUG --- status code --- 05:55:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:50 DEBUG --- http response --- 05:55:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:50 DEBUG ----- output ----- 05:55:50 DEBUG { 05:55:50 DEBUG "status": "success", 05:55:50 DEBUG "data": { 05:55:50 DEBUG "resultType": "vector", 05:55:50 DEBUG "result": [ 05:55:50 DEBUG { 05:55:50 DEBUG "metric": { 05:55:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:50 DEBUG }, 05:55:50 DEBUG "value": [ 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG [...] 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:50 DEBUG }, 05:55:50 DEBUG "value": [ 05:55:50 DEBUG 1713761750, 05:55:50 DEBUG "0" 05:55:50 DEBUG ] 05:55:50 DEBUG } 05:55:50 DEBUG ] 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761750" 05:55:50 INFO [http_cmd]: http status code OK 05:55:50 DEBUG --- status code --- 05:55:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:50 DEBUG --- http response --- 05:55:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:50 DEBUG ----- output ----- 05:55:50 DEBUG { 05:55:50 DEBUG "status": "success", 05:55:50 DEBUG "data": { 05:55:50 DEBUG "resultType": "vector", 05:55:50 DEBUG "result": [ 05:55:50 DEBUG { 05:55:50 DEBUG "metric": { 05:55:50 DEBUG "pod": "ds-cts-0" 05:55:50 DEBUG }, 05:55:50 DEBUG "value": [ 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG [...] 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG "pod": "ds-cts-2" 05:55:50 DEBUG }, 05:55:50 DEBUG "value": [ 05:55:50 DEBUG 1713761750, 05:55:50 DEBUG "0" 05:55:50 DEBUG ] 05:55:50 DEBUG } 05:55:50 DEBUG ] 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761750" 05:55:50 INFO [http_cmd]: http status code OK 05:55:50 DEBUG --- status code --- 05:55:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:50 DEBUG --- http response --- 05:55:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:55:50 DEBUG ----- output ----- 05:55:50 DEBUG { 05:55:50 DEBUG "status": "success", 05:55:50 DEBUG "data": { 05:55:50 DEBUG "resultType": "vector", 05:55:50 DEBUG "result": [ 05:55:50 DEBUG { 05:55:50 DEBUG "metric": { 05:55:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:55:50 DEBUG }, 05:55:50 DEBUG "value": [ 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG [...] 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:55:50 DEBUG }, 05:55:50 DEBUG "value": [ 05:55:50 DEBUG 1713761750, 05:55:50 DEBUG "0" 05:55:50 DEBUG ] 05:55:50 DEBUG } 05:55:50 DEBUG ] 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761750" 05:55:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713761750" 05:55:50 INFO [http_cmd]: http status code OK 05:55:50 INFO [http_cmd]: http status code OK 05:55:50 DEBUG --- status code --- 05:55:50 DEBUG --- status code --- 05:55:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:50 DEBUG --- http response --- 05:55:50 DEBUG --- http response --- 05:55:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:50 DEBUG ----- output ----- 05:55:50 DEBUG ----- output ----- 05:55:50 DEBUG { 05:55:50 DEBUG { 05:55:50 DEBUG "status": "success", 05:55:50 DEBUG "status": "success", 05:55:50 DEBUG "data": { 05:55:50 DEBUG "data": { 05:55:50 DEBUG "resultType": "vector", 05:55:50 DEBUG "resultType": "vector", 05:55:50 DEBUG "result": [ 05:55:50 DEBUG "result": [ 05:55:50 DEBUG { 05:55:50 DEBUG { 05:55:50 DEBUG "metric": { 05:55:50 DEBUG "metric": { 05:55:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:50 DEBUG }, 05:55:50 DEBUG }, 05:55:50 DEBUG "value": [ 05:55:50 DEBUG "value": [ 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG [...] 05:55:50 DEBUG [...] 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:50 DEBUG }, 05:55:50 DEBUG }, 05:55:50 DEBUG "value": [ 05:55:50 DEBUG "value": [ 05:55:50 DEBUG 1713761750, 05:55:50 DEBUG 1713761750, 05:55:50 DEBUG "991.916" 05:55:50 DEBUG "0" 05:55:50 DEBUG ] 05:55:50 DEBUG ] 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:50 DEBUG ] 05:55:50 DEBUG ] 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761750" 05:55:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761750" 05:55:50 INFO [http_cmd]: http status code OK 05:55:50 INFO [http_cmd]: http status code OK 05:55:50 DEBUG --- status code --- 05:55:50 DEBUG --- status code --- 05:55:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:50 DEBUG --- http response --- 05:55:50 DEBUG --- http response --- 05:55:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:50 DEBUG ----- output ----- 05:55:50 DEBUG ----- output ----- 05:55:50 DEBUG { 05:55:50 DEBUG { 05:55:50 DEBUG "status": "success", 05:55:50 DEBUG "status": "success", 05:55:50 DEBUG "data": { 05:55:50 DEBUG "data": { 05:55:50 DEBUG "resultType": "vector", 05:55:50 DEBUG "resultType": "vector", 05:55:50 DEBUG "result": [ 05:55:50 DEBUG "result": [ 05:55:50 DEBUG { 05:55:50 DEBUG { 05:55:50 DEBUG "metric": { 05:55:50 DEBUG "metric": { 05:55:50 DEBUG "instance": "10.142.0.107:9100" 05:55:50 DEBUG "pod": "ds-idrepo-0" 05:55:50 DEBUG }, 05:55:50 DEBUG }, 05:55:50 DEBUG "value": [ 05:55:50 DEBUG "value": [ 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG [...] 05:55:50 DEBUG [...] 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG 05:55:50 DEBUG "instance": "10.142.0.43:9100" 05:55:50 DEBUG "pod": "ds-idrepo-2" 05:55:50 DEBUG }, 05:55:50 DEBUG }, 05:55:50 DEBUG "value": [ 05:55:50 DEBUG "value": [ 05:55:50 DEBUG 1713761750, 05:55:50 DEBUG 1713761750, 05:55:50 DEBUG "0.0005333333333453538" 05:55:50 DEBUG "0" 05:55:50 DEBUG ] 05:55:50 DEBUG ] 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:50 DEBUG ] 05:55:50 DEBUG ] 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:50 DEBUG } 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761751" 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761751" 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "instance": "10.142.0.107:9100" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "instance": "10.142.0.43:9100" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 1713761751, 05:55:51 DEBUG "39424.76666666666" 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG } 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "instance": "10.142.0.107:9100" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "instance": "10.142.0.43:9100" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 1713761751, 05:55:51 DEBUG "0" 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG } 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713761751" 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 1713761751, 05:55:51 DEBUG "11227525120" 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG } 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713761751" 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 1713761751, 05:55:51 DEBUG "1.0369086048642195" 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG } 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761751" 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 1713761751, 05:55:51 DEBUG "0" 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG } 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761751" 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761751" 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 1713761751, 05:55:51 DEBUG "0" 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG } 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 1713761751, 05:55:51 DEBUG "0" 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG } 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761751" 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761751" 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 1713761751, 05:55:51 DEBUG "0" 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG } 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "pod": "ds-idrepo-0" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "pod": "ds-idrepo-2" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 1713761751, 05:55:51 DEBUG "0" 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG } 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761751" 05:55:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713761751" 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "pod": "ds-cts-0" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "pod": "ds-cts-2" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 1713761751, 05:55:51 DEBUG "0" 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG ] 05:55:51 DEBUG } 05:55:51 DEBUG } 05:55:51 INFO [http_cmd]: http status code OK 05:55:51 DEBUG --- status code --- 05:55:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:51 DEBUG --- http response --- 05:55:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:51 DEBUG ----- output ----- 05:55:51 DEBUG { 05:55:51 DEBUG "status": "success", 05:55:51 DEBUG "data": { 05:55:51 DEBUG "resultType": "vector", 05:55:51 DEBUG "result": [ 05:55:51 DEBUG { 05:55:51 DEBUG "metric": { 05:55:51 DEBUG "instance": "10.142.0.107:9100" 05:55:51 DEBUG }, 05:55:51 DEBUG "value": [ 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG [...] 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG 05:55:51 DEBUG "instance": "10.142.0.43:9100" 05:55:51 DEBUG }, 05:55:52 DEBUG "value": [ 05:55:52 DEBUG 1713761751, 05:55:52 DEBUG "31413" 05:55:52 DEBUG ] 05:55:52 DEBUG } 05:55:52 DEBUG ] 05:55:52 DEBUG } 05:55:52 DEBUG } 05:55:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713761752" 05:55:52 INFO [http_cmd]: http status code OK 05:55:52 DEBUG --- status code --- 05:55:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:52 DEBUG --- http response --- 05:55:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:52 DEBUG ----- output ----- 05:55:52 DEBUG { 05:55:52 DEBUG "status": "success", 05:55:52 DEBUG "data": { 05:55:52 DEBUG "resultType": "vector", 05:55:52 DEBUG "result": [ 05:55:52 DEBUG { 05:55:52 DEBUG "metric": { 05:55:52 DEBUG "pod": "ds-idrepo-0" 05:55:52 DEBUG }, 05:55:52 DEBUG "value": [ 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG [...] 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG "pod": "ds-cts-2" 05:55:52 DEBUG }, 05:55:52 DEBUG "value": [ 05:55:52 DEBUG 1713761752, 05:55:52 DEBUG "0" 05:55:52 DEBUG ] 05:55:52 DEBUG } 05:55:52 DEBUG ] 05:55:52 DEBUG } 05:55:52 DEBUG } 05:55:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761752" 05:55:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713761752" 05:55:52 INFO [http_cmd]: http status code OK 05:55:52 DEBUG --- status code --- 05:55:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:52 DEBUG --- http response --- 05:55:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:52 DEBUG ----- output ----- 05:55:52 DEBUG { 05:55:52 DEBUG "status": "success", 05:55:52 DEBUG "data": { 05:55:52 DEBUG "resultType": "vector", 05:55:52 DEBUG "result": [ 05:55:52 DEBUG { 05:55:52 DEBUG "metric": { 05:55:52 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:52 DEBUG }, 05:55:52 DEBUG "value": [ 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG [...] 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:52 DEBUG }, 05:55:52 DEBUG "value": [ 05:55:52 DEBUG 1713761752, 05:55:52 DEBUG "0" 05:55:52 DEBUG ] 05:55:52 DEBUG } 05:55:52 DEBUG ] 05:55:52 DEBUG } 05:55:52 DEBUG } 05:55:52 INFO [http_cmd]: http status code OK 05:55:52 DEBUG --- status code --- 05:55:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:52 DEBUG --- http response --- 05:55:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:52 DEBUG ----- output ----- 05:55:52 DEBUG { 05:55:52 DEBUG "status": "success", 05:55:52 DEBUG "data": { 05:55:52 DEBUG "resultType": "vector", 05:55:52 DEBUG "result": [ 05:55:52 DEBUG { 05:55:52 DEBUG "metric": { 05:55:52 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:52 DEBUG }, 05:55:52 DEBUG "value": [ 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG [...] 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG 05:55:52 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:52 DEBUG }, 05:55:52 DEBUG "value": [ 05:55:52 DEBUG 1713761752, 05:55:52 DEBUG "0" 05:55:52 DEBUG ] 05:55:52 DEBUG } 05:55:52 DEBUG ] 05:55:52 DEBUG } 05:55:52 DEBUG } 05:56:09 INFO 05:56:09 INFO [loop_until]: kubectl --namespace=xlou top pods 05:56:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:56:09 INFO [loop_until]: OK (rc = 0) 05:56:09 DEBUG --- stdout --- 05:56:09 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8790Mi am-767f6b67b5-qcqz2 7m 8838Mi am-767f6b67b5-vbltz 7m 8164Mi ds-cts-0 15m 5291Mi ds-cts-1 12m 11385Mi ds-cts-2 13m 10624Mi ds-idrepo-0 14m 4010Mi ds-idrepo-1 14m 3994Mi ds-idrepo-2 13m 3984Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1182Mi idm-b85f9d978-wnhxb 4m 1243Mi lodemon-7f68c498d9-gbc5j 4m 69Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1001m 10670Mi 05:56:09 DEBUG --- stderr --- 05:56:09 DEBUG 05:56:17 INFO 05:56:17 INFO [loop_until]: kubectl --namespace=xlou top node 05:56:17 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:56:17 INFO [loop_until]: OK (rc = 0) 05:56:17 DEBUG --- stdout --- 05:56:17 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6239Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 8120Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 11497Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3360Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 71m 0% 8521Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5224Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 11710Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 3140Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 60m 0% 10382Mi 9% gke-xlou-cdm-ds-0f601751-j488 68m 0% 7487Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 76m 0% 16545Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 91m 0% 8046Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 63m 0% 13346Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1081m 6% 20877Mi 35% 05:56:17 DEBUG --- stderr --- 05:56:17 DEBUG 05:56:36 INFO Finished: True 05:56:36 INFO Waiting for threads to register finish flag 05:57:17 INFO Done. Have a nice day! :) 127.0.0.1 - - [22/Apr/2024 05:57:17] "GET /monitoring/stop HTTP/1.1" 200 - 05:57:21 WARNING File /tmp/lodemon_data-20240421_231208/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6481', ...) 127.0.0.1 - - [22/Apr/2024 05:57:26] "GET /monitoring/process HTTP/1.1" 200 -